OpenStack GPU Scheduling: How Traits and Flavors Place Specialized Hardware
Ruchi RoyAuthor
Learn how OpenStack uses Placement traits, resource classes, flavors, and PCI passthrough to schedule GPUs and other specialized hardware.
Provisioning a general-purpose virtual machine is relatively straightforward. The scheduler needs to find a host with enough CPU, memory, and storage.
GPU passthrough introduces a harder question: does the host have the right accelerator, connected and configured in the right way, with the NUMA layout the workload needs?
A request for "one GPU" may be incomplete. The application might require a particular GPU family, direct PCI access, or a compatible NUMA layout. Treating every accelerator as interchangeable can lead to failed launches, inconsistent performance, and expensive hardware sitting unused.
We look ar the most common high-intent path: assigning a full physical GPU to a virtual machine through PCI passthrough. Bare metal and vendor virtual-GPU approaches appear later as comparisons, since each follows a different model.
OpenStack coordinates passthrough through several connected mechanisms:
Placement tracks available resources and the characteristics of the providers supplying them.
Resource classes describe consumable inventory.
Traits describe qualitative capabilities.
Flavors turn workload requirements into scheduling requests.
Nova coordinates scheduling and, where configured, attaches PCI devices to virtual machines.
Ironic extends a resource-aware model to bare-metal servers.
Together, these components give operators a structured way to expose specialized infrastructure without asking users to understand the physical layout of every compute node.
Why CPU and memory are not enough
Traditional scheduling starts with quantities: how many virtual CPUs, how much RAM, and how much disk does an instance need?
Specialized hardware adds attributes that capacity alone cannot capture. Two compute nodes may each have an available GPU, yet those GPUs may differ by model, generation, memory, interconnect, driver requirements, or assignment method.
A resource class represents something consumable. Standard examples include virtual CPUs and memory, while operators can define custom resource classes for specialized inventory.
A trait represents a qualitative characteristic of a resource provider. A trait can indicate that a provider has a specific capability, and the trait itself is not consumed.
That distinction matters. Capacity answers, "How much is available?" Traits answer, "What kind of provider is this?"
How flavors connect users to Placement
Most cloud users should not need to know which rack contains a GPU or which compute host has the correct PCI device. They should request a flavor that represents the outcome they need.
In OpenStack, flavor extra specifications shape the request the scheduler evaluates. A flavor can request resources, name a permitted PCI device through an alias, and require or forbid particular traits. Nova then uses that request to identify suitable hosts before applying the remaining scheduling and claim steps.
Conceptually, the relationship looks like this:
Scheduling questionOpenStack mechanism
This abstraction also makes the service catalog clearer. Instead of publishing a vague "GPU instance," an operator can offer workload-oriented flavors whose requirements map to a defined hardware profile.
The important operational discipline is to keep those definitions accurate. A flavor name is a promise. If the underlying inventory, alias configuration, PCI setup, or NUMA design does not support that promise, the scheduler cannot correct the mismatch.
What happens when a GPU-backed instance is launched
The precise implementation varies by cloud, but the scheduling path generally follows five stages.
1. The infrastructure is discovered and described
Compute nodes report their standard inventory to OpenStack. Operators configure the specialized devices that Nova may make available and describe relevant provider capabilities through traits and resource classes.
For bare metal, Ironic nodes can also be associated with a resource class and a set of traits. The corresponding resource-provider information is made available to Placement.
2. The flavor expresses the workload requirement
The selected flavor requests the necessary CPU, memory, storage, accelerator resources, traits, and other relevant policies. For PCI passthrough, the flavor can request a configured PCI alias representing a permitted device type.
3. Placement narrows the candidate set
Placement evaluates providers against the requested inventories and traits. Hosts that cannot meet the resource request—or that lack a required characteristic—are excluded.
This prevents the scheduler from treating every compute node as equivalent when the workload clearly requires specialized hardware.
4. Nova evaluates the remaining constraints
Nova considers the candidate providers alongside other scheduling requirements. Depending on the environment, these may include NUMA affinity, PCI availability, network attachment, availability zones, or operational policies.
5. The resource is claimed and the workload is created
Once a valid destination is selected, the relevant resources are claimed. With full PCI passthrough, the physical device is assigned to one guest and is not shared with other guests while attached.
This final point is important for capacity planning: a PCI-passthrough GPU is dedicated inventory. A cloud cannot safely schedule it as though it were an infinitely divisible resource.
Why traits are a strong fit for specialized hardware
OpenStack can also use host aggregates and aggregate metadata to influence scheduling. The AggregateInstanceExtraSpecsFilter remains a supported mechanism for matching flavor properties to host-aggregate metadata.
However, aggregates and traits model different things.
Aggregates group hosts and remain useful for purposes such as operational segmentation, availability-zone design, licensing boundaries, or groups of hosts that share a policy. Traits describe capabilities on resource providers and participate directly in Placement requests.
For specialized hardware, traits offer several advantages:
They make the requested capability explicit.
They keep qualitative requirements close to the provider that offers them.
They work alongside resource inventory and allocation data in Placement.
They can describe differences within a broader pool of compute or bare-metal nodes.
They allow flavors to require a capability without exposing physical host names to users.
This does not mean every aggregate-based design should be replaced. It means operators should avoid using host grouping as a substitute for accurate resource modeling. A sound design uses each mechanism for the problem it represents.
GPU virtual machines and GPU bare metal solve different problems
There is no single correct way to expose accelerators.
With PCI passthrough, a virtual machine receives direct access to a physical GPU or another PCI device. This dedicates the device to that guest and introduces hardware, driver, IOMMU, NUMA, and lifecycle requirements.
With bare metal, Ironic provisions the physical server itself. Resource classes can identify a class of GPU-equipped nodes, while traits can express more specific qualitative requirements. This model is useful when a workload needs the entire server, maximum hardware control, or a topology that should not be abstracted through a hypervisor.
Vendor-specific GPU partitioning or virtual GPU approaches introduce another set of decisions. Their availability depends on the accelerator, drivers, firmware, licensing, hypervisor support, and deployment design. They should not be treated as interchangeable with full PCI passthrough.
The right question is therefore not simply, “Does the cloud support GPUs?” It is, “Which GPU delivery model matches this workload, and can the platform operate it reliably?”
Networking is part of the placement decision
A GPU may be the most visible specialized resource, but it rarely works in isolation. Distributed training and other high-performance workloads can depend on NIC capabilities, PCI topology, NUMA locality, packet-processing design, and storage throughput.
Technologies such as SR-IOV, DPDK, and hardware offload address different parts of this problem:
SR-IOV allows a supported physical device to expose virtual functions that can be assigned to guests.
DPDK enables user-space packet processing designed for high-throughput and low-latency networking.
ASAP² is associated with supported NVIDIA/Mellanox hardware and offloads Open vSwitch data-plane processing to the NIC.
These are not generic performance switches. Results depend on the specific NIC, firmware, drivers, BIOS configuration, network design, and operating model. A design validated for one hardware vendor should not be assumed to behave identically on another.
That is why specialized-hardware planning must begin before the cloud is deployed. The compute, network, storage, and scheduling models need to agree on what the service is promising.
A practical design checklist
Before publishing GPU or accelerator-backed flavors, answer these questions:
Which workloads are being served? Training, inference, visualization, network functions, and HPC workloads can have very different requirements.
Does the workload need a full device, a virtual function, or an entire bare-metal server?
Which hardware distinctions affect compatibility or performance? Identify only the resource classes and traits that users genuinely need.
Does each flavor map to verified inventory? Avoid names or descriptions that promise capabilities the underlying hosts do not consistently provide.
Have PCI and NUMA relationships been considered? A valid device allocation is not automatically an optimal topology.
Does the network match the compute design? Validate the NIC, firmware, drivers, switching, and offload approach as one system.
Can the platform operate the service over time? Monitoring, capacity visibility, upgrades, hardware replacement, and failed-device handling are part of the product—not afterthoughts.
The best flavor catalog is not the one with the most options. It is the one whose options map cleanly to real, supportable infrastructure.
Where Atmosphere fits
Atmosphere supports GPUs and PCI passthrough across virtual machines, bare metal, and Kubernetes, and it is available through hosted, on-premises, support, and managed engagement options.
GPU work rewards getting the details right: the specific GPU model, whether a workload needs a full-card passthrough or a partitioned device, the NUMA topology, the surrounding networking, and how the hardware is operated over its lifecycle. Those questions are worth working through before a design becomes a production constraint.
If you are planning GPU infrastructure, you can bring your GPU model, passthrough or partitioning requirements, NUMA topology, networking, and lifecycle needs to VEXXHOST to work through together.
Share
Virtual machines, Kubernetes & Bare Metal Infrastructure
Choose from Atmosphere Cloud, Hosted, or On-Premise. Simplify your cloud operations with our intuitive dashboard. Run it yourself, tap our expert support, or opt for full remote operations. Leverage Terraform, Ansible or APIs directly powered by OpenStack & Kubernetes
