
Air gapped cloud upgrades with Atmosphere release manifests
Air gapped cloud on OpenStack: how Atmosphere's digest-pinned release manifests move a release across the boundary, and what the manifest leaves out.
Read field noteField notes / Latest
Engineering notes from operating open infrastructure: the failures, design decisions, and upstream work that make open infrastructure better.
Browse all field notes
Air gapped cloud on OpenStack: how Atmosphere's digest-pinned release manifests move a release across the boundary, and what the manifest leaves out.
Read field note
Assess enterprise AI readiness before investing in GPUs. Explore 12 key factors covering data, models, infrastructure, governance, operations, and costs.
Read field note
Explore on-premises Kubernetes architecture, costs, and operational challenges. Learn when managed Kubernetes makes sense for production workloads.
Read field noteHow the OpenStack hypervisor stack fits together: Nova virt drivers, libvirt, QEMU and KVM, with nova.conf keys and /dev/kvm checks.
Run openstack hypervisor list on a cloud whose compute nodes use hardware-accelerated KVM, and the Hypervisor Type column says QEMU. Run it on a lab cloud that translates every guest instruction in software, and it also says QEMU. The OpenStack hypervisor field reports a libvirt driver name, so it cannot tell you whether a guest is using VT-x or AMD-V. Answering that means walking down four layers: Nova's virt driver, libvirt, the QEMU process and the KVM kernel module. Version references below are to Nova 33.0.2, the current 2026.1 Gazpacho release.
Take one instance as it lands on a host. The nova-compute service there hands the request to its configured virt driver, which translates operations such as spawn, reboot, live migration and resize into calls on a hypervisor already installed on the node. With libvirt.LibvirtDriver, Nova builds a libvirt domain XML document for the instance and sends it to libvirtd over the URI in [libvirt] connection_uri. The first line of that document already names the hypervisor, as <domain type='kvm'> or <domain type='qemu'>.
libvirtd then starts the guest. Its QEMU driver runs each virtual machine as its own QEMU process, and under SELinux sVirt each of those processes gets its own confined label. QEMU is the userspace program that models the machine, with its vCPUs, memory, disks and virtio devices. Its documentation lists KVM as one of several accelerators, next to the Tiny Code Generator (TCG), a JIT that emulates CPUs in software.
For a kvm domain, the QEMU process opens /dev/kvm, the device exposed by the Linux kernel module (kvm, plus kvm_intel or kvm_amd), and issues KVM_CREATE_VM and KVM_CREATE_VCPU ioctls. From there, guest code runs directly on the physical CPU.
KVM lives in the kernel and QEMU lives in userspace, and that split decides who patches what. A KVM flaw such as CVE-2026-53359 is fixed by a host kernel update and a reboot plan for each compute node. A QEMU fix arrives as a userspace package. Neither one touches the OpenStack control plane.

[DEFAULT] compute_driver has no default value, so every nova-compute sets it explicitly. Our OpenStack compute service runs virtual machines on KVM hypervisors. Its feature list marks multi-architecture compatibility, including x86 and Arm, for all editions, and our public cloud offers 64-bit Arm-based chips alongside Intel x86. Atmosphere 7.8.1 deploys Nova from images tagged 2025.2 (Flamingo), pinned in roles/defaults/vars/main.yml, and the driver list, the virt_type choices and the defaults quoted in this post are identical in Nova 32.2.1, the latest 2025.2 release. Nova 33.0.2 accepts five in-tree drivers, listed in the hypervisors guide:
libvirt.LibvirtDriver for Linux hosts. [libvirt] virt_type picks the backend and accepts kvm (the default), qemu, lxc and parallels. Since Nova 22.0.0, upstream considers only kvm and qemu supported, because the other backends have no upstream testing.ironic.IronicDriver for bare metal, covered below.vmwareapi.VMwareVCDriver for vCenter. Its release note marks it experimental: "not tested by the OpenStack project and does not have a clear maintainer."zvm.ZVMDriver for z/VM on IBM Z and LinuxONE.fake.FakeDriver, which spawns nothing and exists for tests.Older guides still list drivers that are gone. Hyper-V was removed in Nova 29.0.0 (2024.1), and the RDP console APIs that only it used now return HTTP 400. PowerVM was removed in Nova 26.0.0 (Zed), and the libvirt driver's Xen backend was dropped in Nova 23.0.0.
The upstream KVM guide enables KVM explicitly with two keys:
[DEFAULT]
compute_driver = libvirt.LibvirtDriver
[libvirt]
virt_type = kvmNova copies virt_type into the type attribute of the root <domain> element for every guest it defines (driver.py, guest.virt_type = CONF.libvirt.virt_type), and libvirt's domain format uses that attribute to choose the hypervisor that runs the guest. The option help lists five settings that depend on it: connection_uri, disk_prefix, cpu_mode, cpu_models and tb_cache_size.
With virt_type set to kvm or qemu, an unset cpu_mode defaults to host-model for x86_64 guests on x86_64 hosts, which clones the host CPU's feature flags into the guest. For AArch64 guests on AArch64 hosts, Nova's _get_guest_cpu_model_config picks host-passthrough, because libvirt and QEMU cannot describe an Arm host CPU model. A guest emulated for another architecture through hw_emulation_architecture loses that mode: _get_guest_cpu_config clears it and, for AArch64, sets the cortex-a57 model. host-passthrough gives the guest the exact host CPU model. custom pins a named model from cpu_models, and then mitigation flags such as md-clear become your job; our OpenStack security baseline shows that cpu_model_extra_flags configuration.
In Atmosphere 7.8.1 we leave both virt_type and cpu_mode unset in roles/nova/vars/main.yml, so x86 compute nodes run Nova's defaults, kvm and host-model. Our Nova chart sets compute_driver = libvirt.LibvirtDriver and points connection_uri at qemu+unix:///system?socket=/run/libvirt/libvirt-sock (charts/nova/values.yaml). That socket belongs to libvirtd, which runs as a Kubernetes DaemonSet from our ghcr.io/vexxhost/libvirtd:2025.2 image. The same role sets live_migration_scheme = tls and swtpm_enabled = true, which turns on emulated TPM devices for guests.
openstack hypervisor list say QEMU?Here is the command on python-openstackclient 10.3.0. The column headers match the client's source; IDs, hostnames and addresses are illustrative.
openstack hypervisor list+--------------------------------------+------------------------+-----------------+-----------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+--------------------------------------+------------------------+-----------------+-----------+-------+
| 0b8f2c3e-6a51-4c1e-9f0d-2f6f3b7c1a10 | compute-01.example.com | QEMU | 10.0.0.21 | up |
| 5d41e7a9-93c2-4b8e-a0f4-7e2c9d1b6f22 | compute-02.example.com | QEMU | 10.0.0.22 | up |
+--------------------------------------+------------------------+-----------------+-----------+-------+The libvirt driver fills hypervisor_type from libvirt's virConnectGetType(), and the libvirt API documentation says plainly that "both KVM and QEMU guests are serviced by the driver for the qemu:// URI, so a return of "QEMU" does not indicate whether KVM acceleration is present." hypervisor_version comes from virConnectGetVersion(), encoded as major × 1,000,000 + minor × 1,000 + release, so 8002002 means QEMU 8.2.2:
openstack hypervisor show compute-01.example.com \
-c hypervisor_type -c hypervisor_version -c state -c status+--------------------+---------+
| Field | Value |
+--------------------+---------+
| hypervisor_type | QEMU |
| hypervisor_version | 8002002 |
| state | up |
| status | enabled |
+--------------------+---------+Nodes managed by the Ironic driver report ironic in the same column.
Capacity figures have left this API. Compute microversion 2.88 removed vcpus, memory_mb, local_gb, running_vms and the other usage fields, because they were "frequently misleading or outright wrong." openstacksdk 4.19.0 sends hypervisor requests at the lower of 2.88 (the Hypervisor resource's _max_microversion) and the cloud's maximum (2.100 on Flamingo, 2.103 on Gazpacho), unless you pin a microversion with --os-compute-api-version. At 2.88, python-openstackclient hides those fields and --long adds no columns. A pinned microversion below 2.88 makes supports_microversion return false, and the old columns come back. Placement holds the numbers, and the osc-placement plugin (4.9.1) reads them:
openstack resource provider list --name compute-01.example.com
openstack resource provider inventory list <provider-uuid> \
-c resource_class -c allocation_ratio -c total -c used+----------------+------------------+--------+--------+
| resource_class | allocation_ratio | total | used |
+----------------+------------------+--------+--------+
| VCPU | 4.5 | 64 | 210 |
| MEMORY_MB | 0.9 | 515794 | 262144 |
+----------------+------------------+--------+--------+The totals and usage are illustrative, and the output is abridged to two rows. Atmosphere 7.8.1 sets both ratios shown, cpu_allocation_ratio: 4.5 and ram_allocation_ratio: 0.9, in roles/nova/vars/main.yml. When either option is set, Nova overwrites the Placement inventory's ratio with it. Placement computes capacity as int((total - reserved) * allocation_ratio), and the libvirt driver reports VCPU reserved from [DEFAULT] reserved_host_cpus, which defaults to 0. With 0 reserved, those 64 physical threads can be allocated as 288 vCPUs.
The API can't tell you, so check the host. These commands follow the Nova KVM guide and libvirt's own tooling:
grep -E -c 'svm|vmx' /proc/cpuinfo # non-zero: CPU has VT-x or AMD-V (firmware may still disable it)
lsmod | grep kvm # expect kvm_intel or kvm_amd
stat -c '%U:%G %a' /dev/kvm # expect group kvm; Atmosphere's libvirt pod sets mode 660
virt-host-validate qemuvirt-host-validate prints one line per check. From its source, the KVM lines on a healthy x86 host are (abridged; the full run also covers cgroups and IOMMU):
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking if device '/dev/kvm' exists : PASS
QEMU: Checking if device '/dev/kvm' is accessible : PASSlibvirt's QEMU driver offers KVM guests only when /dev/kvm exists and the user QEMU runs as can read and write it (virQEMUCapsKVMUsable), so device permissions matter. Nova's guide warns that loading the KVM module before installing nova-compute can leave /dev/kvm without the kvm group, and that udevadm trigger fixes it. In Atmosphere, the libvirt container's start script (_libvirt.sh.tpl) runs chmod 660 and chown root:kvm on /dev/kvm each time it starts.
The definitive check is the guest's own domain XML. Domain names follow instance_name_template, which defaults to instance-%08x, and an admin can read one with openstack server show <server> -c OS-EXT-SRV-ATTR:instance_name:
virsh dumpxml instance-0000002a | head -n 1
# On Atmosphere, virsh runs inside the libvirt pod on that node:
kubectl -n openstack get pods -l application=libvirt -o wide
kubectl -n openstack exec <libvirt-pod> -c libvirt -- virsh dumpxml instance-0000002a | head -n 1If nova-compute.log shows libvirtError: internal error no supported architecture for os type 'hvm', the Nova troubleshooting section says the KVM kernel modules are not loaded.
Labs often start on TCG and later move to hardware with VT-x or AMD-V. Editing virt_type changes nothing for guests already defined, because their domain XML still carries type='qemu', and Nova reads the option only when nova-compute starts. Once the host checks above pass, set virt_type = kvm under [libvirt], restart nova-compute, and hard reboot each guest on the host. Each hard reboot stops the guest, so plan it as guest downtime, host by host; when a whole fleet needs the change, our engineers can help plan the rollout as part of the Day 2 work our professional services cover. The libvirt driver's _hard_reboot destroys the domain and regenerates its XML from current configuration, so the new type takes effect there. A successful soft reboot calls shutdown() and launch() on the existing domain and keeps the old type. Nova falls back to a hard reboot only when the soft reboot fails, so pass --hard explicitly:
openstack server list --all-projects --host compute-01.example.com -f value -c ID
openstack server reboot --hard <server-id>
virsh dumpxml <instance-name> | head -n 1 # now opens with <domain type='kvm'Nova's QEMU page is direct: QEMU without KVM "has worse performance than KVM and is a poor choice for a production deployment." It names two uses, older hardware without virtualization extensions and nova-compute running inside a VM for development or testing.
TCG has one tuning knob worth knowing. QEMU 5.0 raised the default translation block cache from 32 MiB to 1 GiB per guest, and [libvirt] tb_cache_size (libvirt 8.0.0 or later) shrinks it so more emulated guests fit in a small test host's memory.
Nova also falls back to TCG by itself on a kvm host. When an image sets hw_emulation_architecture=aarch64 with hw_firmware_type=uefi and hw_machine_type=virt, the libvirt driver sets that guest's domain type to qemu because the guest and host architectures differ (_check_emulation_arch, available since Nova 25.0.0). An Arm image then boots on x86 hardware, slowly, provided the compute node has the qemu-system-XXX and qemu-efi-aarch64 packages that Nova's emulation guide requires. Emulation is avoidable when Arm hardware is available. When an image sets hw_architecture=aarch64, Nova's ImagePropertiesFilter places it only on a host that advertises that architecture, and on an Arm host the guest and host architectures match, so the libvirt driver keeps the kvm domain type and has nothing to emulate. An x86 host can also advertise AArch64 guest support through libvirt, because Nova's _get_instance_capabilities lists the guest architectures libvirt reports, so with ImagePropertiesFilter enabled, hw_architecture=aarch64 restricts placement to native AArch64 hosts only when no other eligible host advertises AArch64 guest support. Images without hw_architecture fall back to image_properties_default_architecture, which Atmosphere 7.8.1 sets to x86_64 in roles/nova/vars/main.yml.
If your compute host is itself a VM, nested KVM usually beats TCG. The outer host loads kvm_intel or kvm_amd with nested=1, the kernel default since Linux 4.20 unless a distribution overrides it (cat /sys/module/kvm_intel/parameters/nested prints Y, per the kernel docs). When the VM that runs nova-compute is itself a Nova instance, the outer cloud's cpu_mode must be host-passthrough, or host-model or custom with vmx exposed (cpu_model_extra_flags = vmx). Nova's guide names vmx, the Intel flag; on AMD hosts the virtualization flag is svm (libvirt's flag check). Our Atmosphere quick start warns that running it inside VMs without nested virtualization leaves guest performance "un-usable." The two sources describe live migration differently. Nova's KVM guide says that while nested guests run, the outer guest fails to complete live migration and fails to resume from suspend. The kernel's v6.16 guide says migrating an outer guest with a running nested guest works as of Linux 5.3 and QEMU 4.2.0 on Intel x86, and that on AMD the outer guest should not be migrated or saved until the nested guest shuts down. On a multi-tenant cloud it is also an isolation decision. Nested virtualization is available in every edition of our compute service, and in our CVE-2026-53359 write-up we recommend treating it as an intentional capability, exposed only where the use case requires it.
When a workload needs the whole server, ironic.IronicDriver takes the hypervisor out of the path. According to Nova's Ironic page, one nova-compute service manages many Ironic nodes, each node becomes a compute node whose hypervisor_hostname is the Ironic node UUID, and the instance to compute node to Ironic node mapping is 1:1:1. Scheduling runs on custom resource classes. Flavor RAM and CPU values are ignored, and the Ironic flavor guide zeroes the standard resources:
openstack flavor set --property resources:CUSTOM_BAREMETAL_WITH_GPU=1 my-baremetal-flavor
openstack flavor set --property resources:VCPU=0 my-baremetal-flavor
openstack flavor set --property resources:MEMORY_MB=0 my-baremetal-flavor
openstack flavor set --property resources:DISK_GB=0 my-baremetal-flavorFor dedicated accelerators, our GPU scheduling post compares PCI passthrough on KVM with GPU-equipped Ironic nodes and explains how flavors, traits and resource classes steer scheduling for each.
The trade is lifecycle operations. Nova lists migrate, resize, snapshot, pause, shelve and evacuate as missing for Ironic instances, and it reports known issues past 1,000 Ironic nodes on a single nova-compute process. Our compute service offers bare metal provisioning, live migration and custom flavors in its Hosted and On-Premise editions. So the choice comes down to what tenants expect from the server after it boots. If they expect to resize, snapshot or evacuate it, keep them on KVM guests. If the workload needs the whole machine and can live without those operations, an Ironic flavor gives them the hardware with no hypervisor in the path.
Choose from Atmosphere Cloud, Hosted, or On-Premise.
Simplify your cloud operations with our intuitive dashboard.
Run it yourself, tap our expert support, or opt for full remote operations.
Leverage Terraform, Ansible or APIs directly powered by OpenStack & Kubernetes