Field notes / Latest
Engineering notes from operating open infrastructure: the failures, design decisions, and upstream work that make open infrastructure better.
Browse all field notes
Learn the difference between Canadian sovereign cloud and data residency, including jurisdiction, infrastructure control, provider ownership, private cloud, and OpenStack.
Read field note
Evaluate OpenStack as a VMware alternative, including networking, storage, hardware reuse, migration methods, downtime, workload waves, and Day 2 operations.
Read field note
Infrastructure exceptions add hidden complexity to automation, upgrades, troubleshooting, and operations. Learn when standardization matters.
Read field note
Booting a GPU server isn't the same as making it production-ready. How Atmosphere unifies OpenStack, Ironic, and Kubernetes to turn H200 hardware into recoverable, reusable AI infrastructure.
An AI cloud is ready only when its control plane can turn a request into usable, recoverable and reusable accelerator capacity. This article connects all of those responsibilities so the result cannot be mistaken for a one-off server boot. It establishes the top-level evidence that Atmosphere is ready to operate the validated H200 AI-infrastructure path, while identifying the parts that remain separate or still need acceptance.
This effort was not a test of one OpenStack service or one server boot method. We set out to validate Atmosphere as the complete operating platform for physical GPU infrastructure: prepare the cloud control plane on Kubernetes, enroll real accelerator hardware, deploy its operating system, attach its networks, expose a usable remote console, recover it from firmware failures, return it to the resource pool, and finally consume the same hardware through a declarative Kubernetes workflow. The infrastructure exists to serve people doing real AI work: interactive development, model training and fine-tuning, batch processing, and online or offline inference.
💡 Major Takeaway: Unified GPU Infrastructure with Atmosphere
Building a production GPU cloud is not about booting individual servers. It requires an integrated platform (Atmosphere) that connects OpenStack, Ironic and Kubernetes to one physical hardware pool. The direct, unvirtualized eight-H200 path passed an AI workload; the declarative mCAPI path passed its physical control-plane slice and still requires full one-control-plane plus two-worker acceptance.
Both paths share the same cloud and hardware foundations. Atmosphere operates the cloud services on top of Kubernetes; OpenStack represents capacity and the user-facing lifecycle; Ironic translates requests into physical provisioning; and Redfish controls the out-of-band hardware path. The guest image and first-boot system complete configuration inside the installed server. The difference is the consumer contract: Nova gives a user a whole H200 server, while Magnum, mCAPI, and Cluster API reconcile a Kubernetes cluster request into the Nova and Ironic machines that become its control-plane and worker nodes. Both paths return usable physical GPU capacity to the same AI workloads.
Atmosphere provides an opinionated platform to deploy, operate, and reconcile OpenStack services containerized on Kubernetes. Ironic adds physical-node provisioning to that cloud. The combination is particularly powerful for AI workflows because the same Atmosphere-managed control plane can serve both direct bare-metal compute and containerized Kubernetes worker nodes while physical mechanics remain reliably delegated to Ironic.
Atmosphere ensures the control plane's cross-service contracts remain aligned:
Breaking any one of these contracts can leave the cloud healthy while physical deployment fails. Atmosphere's declarative deployment model keeps these contracts predictable across updates and operational lifecycles.
The cloud and hardware layers don't deliver the final product. Different users consume the same physical capacity in different ways:
These needs explain why a successful server boot is not enough. A useful handoff must provide a stable operating system, accelerator visibility, access to the required model and data paths, working network connectivity, and an operational recovery path. From there, workload acceptance can run an actual training, fine-tuning, or inference scenario and measure whether the environment meets that workload's functional and performance objectives.
Installing accelerators is only the first step in building a GPU cloud. The harder problem is turning each physical server into capacity that can be requested through an API, deployed repeatedly, observed during failure, and returned safely to the resource pool.
Atmosphere provides the overarching architecture and operational engine to unify these systems into a cohesive production platform:
We assembled this path and validated its physical lifecycle on NVIDIA H200 hardware powered by Atmosphere. The useful result was not simply that the server booted. The result was a repeatable chain from cloud API request to local-disk deployment, networked guest, remote graphical console, cleanup, and resource reuse.
A physical server differs from a virtual machine in one important way: many failures happen outside the hypervisor boundary. Firmware can retain stale boot state. A BMC can accept a request but not apply it. A network interface may appear in a different mode after a firmware reset. An operating-system image can boot successfully yet fail first-boot networking.
The architecture therefore needs explicit fault domains:
Operators should be able to inspect every boundary independently. A Nova server in ACTIVE state is useful evidence, but it does not prove that the guest has configured its network. An Ironic node in active state proves that deployment completed from Ironic's perspective, but it does not prove application readiness.
With Atmosphere, the base cloud is deployed and validated in a strict dependency order on Kubernetes before adding Ironic:
This order keeps failures attributable. If Kubernetes storage is unhealthy, debugging a conductor rollout adds noise. If Glance cannot serve an image, testing Redfish media insertion is premature.
It also supports smaller reconciliations. Atmosphere allows changes to a console image or service configuration without requiring a full cloud rebuild. A new guest image can be promoted through Glance without disrupting the control plane. Every deployable artifact uses an immutable version or digest so rollback identifies an exact prior state.
This is why a GPU cloud built with Atmosphere is an operational system rather than a static hardware inventory. Atmosphere makes the underlying physical lifecycle observable, automated, and repeatable.
The same Atmosphere architecture applies to hosted and on-premises deployments, but the integration details remain local:
Those decisions should be explicit before enrollment. Atmosphere provides the cloud platform and resource model, but it cannot remove the need for accurate physical inventory and operating procedures.
Three practices mattered most:
With those practices, Atmosphere provides a strong foundation for operable GPU infrastructure.
If you're interested in learning more, reach out to us for a conversation.
Choose from Atmosphere Cloud, Hosted, or On-Premise.
Simplify your cloud operations with our intuitive dashboard.
Run it yourself, tap our expert support, or opt for full remote operations.
Leverage Terraform, Ansible or APIs directly powered by OpenStack & Kubernetes
