A GPU cloud is only as reliable as the control plane underneath it. Seven staged verification gates, from Kubernetes health to physical acceptance testing, that isolate real hardware failures from software bugs before they reach production.
Every H200 lifecycle depends on a healthy Kubernetes and OpenStack foundation. This article makes that foundation testable instead of implicit: each gate has an owner, an observable result and a stopping condition.
A reliable bare-metal GPU cloud starts long before the first physical compute node is enrolled. Controller preparation, Kubernetes health, storage readiness, image provenance, and network reachability dictate whether physical provisioning via Ironic will succeed or fail later.
💡 Major Takeaway: Isolating the "Ghost Hardware Bug"
Never debug physical bare-metal hardware on an unvalidated control plane. In GPU infrastructure, control-plane misconfigurations (database timeouts, missing Glance artifacts, bad Neutron routes) inevitably manifest as apparent physical BMC or hardware failures. By enforcing strict, staged contract verification (Host → Kubernetes → OpenStack Services → Ironic → Network Reachability) in Atmosphere, infrastructure teams eliminate false hardware diagnostics and isolate software bugs before touching high-value GPU servers.
A cloud user authenticates to the published OpenStack APIs, selects an approved image, flavor, and network, requests a physical GPU server, reaches the installed guest, and later deletes it without needing BMC credentials or knowledge of the provisioning workflow. Controller readiness is valuable only when it can deliver that lifecycle reliably.
The safest build method is staged. Each stage enforces a strict contract gate, produces verifiable evidence, and can be repeated without guessing which previous step completed.
The 7 verification gates progress in dependency order:
Stage 1: Gate 1 – Establish The Host Baseline
Prepare controllers one at a time. Record hardware inventory before changing disks or network configuration. At minimum, verify:
- management access and passwordless administrative automation;
- stable hostnames and name resolution;
- time synchronization;
- persistent interface names and link state;
- storage devices and intended storage ownership;
- BMC reachability and certificate policy; and
- required routes between controllers, service networks, and BMCs.
Do not begin a distributed control-plane deployment while controller identity or storage ownership is ambiguous.
Stage 2: Gate 2 – Build Kubernetes And Storage Foundation
Atmosphere runs OpenStack services containerized on Kubernetes, so Kubernetes health is a hard dependency rather than an implementation detail. Validate the platform before deploying OpenStack:
kubectl get nodes
kubectl get pods --all-namespaces
kubectl get storageclass
Every controller expected to schedule control-plane workloads should report Ready. Investigate Pending, CrashLoopBackOff, or repeatedly restarting system pods before continuing.
If the deployment uses Ceph, verify quorum, manager availability, and storage health independently. An OpenStack pod becoming Ready does not prove that its persistent data path is healthy.
Stage 3: Gate 3 – Deploy The Base OpenStack Services
Deploy identity, image, network, compute, dashboard, and supporting services before Ironic. The exact command depends on the Atmosphere release and automation wrapper, but the acceptance gate is portable:
openstack endpoint list
openstack service list
openstack compute service list
openstack network agent list
openstack image list
Confirm that:
- API endpoints resolve from the operator path;
- database migrations completed;
- message-queue clients are stable;
- Nova conductors and schedulers are up;
- Neutron agents or controllers match the intended backend; and
- Glance can store and return an image.
This gate keeps Ironic failures from being confused with a broken base cloud.
Stage 4: Gate 4 – Reconcile Ironic Services
Ironic adds several specific control-plane responsibilities:
- API and conductor services;
- Nova compute services using the Ironic driver;
- provisioning network access;
- deploy-agent kernel and ramdisk artifacts;
- boot-image publication reachable by BMCs;
- console proxy services when graphical console is enabled; and
- RBAC allowing per-node console workloads where applicable.
After deployment, inspect both Kubernetes and OpenStack:
kubectl get pods --all-namespaces
openstack baremetal conductor list
openstack compute service list --service nova-compute
openstack baremetal driver list
The driver list should expose the interfaces required by the hardware, and the conductor responsible for a node must be online before enrollment proceeds.
Stage 5: Gate 5 – Promote Immutable Images
Bare-metal deployment involves more image types than a typical virtual-machine workflow:
- deploy kernel;
- deploy ramdisk;
- UEFI bootloader or ESP artifact when required;
- instance image;
- service container images; and
- optional console-container images.
Record checksums and immutable container digests before promotion. Verify that the running workloads consume the configured versions. Mutable tags make rollback and incident comparison unreliable.
For instance images, validate format, checksum, architecture, boot mode, and cloud-init support. A large GPU guest image should not be the first artifact used to test the deployment path; first prove the lifecycle with the smallest image that still represents the intended boot and first-boot behavior.
Stage 6: Gate 6 – Validate Physical Reachability Across Boundaries
The control plane needs distinct reachability across explicit boundaries:
Ironic conductor -> BMC API
BMC -> virtual-media publisher
deploy agent -> image source and Ironic callbacks
guest -> intended tenant or provider network
operator -> API and console endpoints
cloud user -> Nova or Horizon -> guest and workload endpoint
Test each direction from the component that initiates the connection. Testing a BMC address from an operator laptop does not prove that the conductor can reach it. Testing the media URL from a controller does not prove that the BMC accepts the URL, certificate, or HTTP behavior.
Stage 7: Gate 7 – Run A Physical Acceptance Test
Use a dedicated node, resource class, flavor, image, and network contract. Run the acceptance through the same Nova or Horizon path offered to cloud users. The acceptance should include:
- node validation and transition to available;
- a cloud user submitting a server request from the published resources;
- Nova scheduling to the intended physical node;
- deploy-agent boot and callback;
- local-disk image write and reboot;
- Nova ACTIVE and Ironic active;
- first-boot completion;
- cloud-user guest access and network reachability;
- graphical-console verification when enabled; and
- user-requested deletion and return to available.
Capture the server, node, port, and service states before cleanup. Cleanup is part of the test, not an administrative afterthought. A representative AI workload remains a separate workload-specific acceptance after the cloud has handed over usable capacity.
On-Premises And Hosted Variants
This workflow works for hosted and customer-owned infrastructure, but the hardware contract changes by deployment:
- interface naming and switch-port ownership;
- BMC certificate and network policy;
- firmware update responsibility;
- storage layout;
- image-publisher location; and
- maintenance-window authority.
Keep those values in deployment inventory, not in reusable automation logic. The automation should consume a clearly reviewed contract.
Recovery Rules
When a stage fails:
- stop at the first failed gate;
- preserve the failing state and relevant logs;
- identify the owning fault domain;
- apply the smallest correction;
- rerun the failed stage and its acceptance gate; and
- avoid broad redeployment until the local correction is proven.
This staged approach turns a complex physical cloud build into a sequence of observable systems. It also makes future upgrades safer because each layer has an independent health contract.
Development: reproduce, test, and contribute
Encode the gates as commands
Start with host and Kubernetes health. The exact storage checks depend on the selected backend, but no OpenStack reconciliation should begin while nodes or required system workloads are unhealthy:
kubectl get nodes -o wide
kubectl wait --for=condition=Ready node --all --timeout=10m
kubectl get pods --all-namespaces --field-selector=status.phase!=Running
kubectl get persistentvolumeclaims --all-namespaces
Then check the OpenStack API boundary with appropriately scoped credentials:
openstack token issue >/dev/null
openstack service list
openstack endpoint list
openstack compute service list
openstack network agent list
openstack image list
openstack baremetal conductor list
Treat expected empty inventories separately from service failures. A new cloud may have no nodes or workload images, but its APIs and conductors must respond.
Render before applying
For an Ansible-managed deployment, perform syntax and inventory checks before the selected playbook:
INVENTORY=${INVENTORY:?set INVENTORY}
PLAYBOOK=${PLAYBOOK:?set PLAYBOOK}
ansible-inventory -i "$INVENTORY" --graph
ansible-playbook -i "$INVENTORY" "$PLAYBOOK" --syntax-check
ansible-playbook -i "$INVENTORY" "$PLAYBOOK" --check --diff
Not every Kubernetes or Helm action supports a side-effect-free Ansible check mode. Review reported skips and render Helm templates explicitly where the role provides that facility. Apply one failed stage at a time, then repeat all gates that depend on it.
A bare-metal GPU cloud earns operational trust when users can request a server, access the installed guest, and release it back into the available pool—and operators can identify where that lifecycle breaks. The seven gates make those expectations testable, with evidence that helps teams isolate failures and verify changes before expanding the deployment.
Keep these gates as acceptance criteria for upgrades, image changes, and new hardware. Passing them establishes a usable infrastructure foundation; representative AI workloads must then confirm that the GPUs, software stack, and data paths meet the intended performance requirements.
VEXXHOST’s GPU infrastructure offering brings Atmosphere and OpenStack to both hosted and customer-owned hardware, with engineering support and optional managed operations. For teams building an H200 cloud, that connects the initial deployment to the ongoing responsibility of keeping GPU capacity accessible, supportable, and ready for reuse.
