Learn how production AI changes infrastructure requirements for availability, recovery, storage, networking, monitoring, capacity, and Day 2 operations.
Running an AI workload is one problem. Operating AI as dependable production infrastructure is another.
During experimentation, infrastructure failure is often inconvenient rather than business-critical. A GPU node goes offline, a training job restarts, or an inference endpoint is temporarily unavailable. The engineering team investigates, restores the workload, and continues.
The consequences change once applications, internal workflows, customer-facing services, or business processes depend on that workload.
“A training job failed” becomes:
“A training job failed after running for three days, and there is no usable checkpoint.”
“An inference server went offline” becomes:
“A customer-facing application depends on that endpoint, and there is no redundant serving capacity.”
At that point, GPU performance is only part of the infrastructure question. Teams also need to consider failure domains, recovery, storage durability, networking, observability, capacity planning, maintenance, and operational ownership.
This is increasingly the infrastructure discussion around production AI: not simply “Can we run it?”, but “Can the business reliably depend on it?”
What Changes When an AI Workload Becomes Business-Critical?
Production readiness depends on the consequence of failure, not simply the type of workload.
Not every AI workload needs high availability. An experimental training job may tolerate several hours of interruption. An internal service may accept occasional downtime. A customer-facing inference endpoint supporting a transaction or operational process may not.
The first step is therefore to define requirements for each workload:
- How much downtime is acceptable?
- How much completed work can be lost?
- Can the workload restart automatically?
- What infrastructure failures should it tolerate?
- What happens to the application during recovery?
- Who owns restoration when something fails?
Once those answers matter to the business, AI has become a production infrastructure problem.
Design Around Failure Domains, Not Just Available GPUs
A resilient AI platform starts by assuming that every infrastructure component can eventually fail.
GPUs can fail. Servers can fail. Storage devices, network interfaces, switches, controllers, and entire racks or sites can fail.
The question is what happens next.
For a critical service, infrastructure teams should understand failure domains across the stack:
GPU → server → rack → network → storage → availability zone or site
Two inference replicas, for example, provide limited protection if both run on the same physical server. Storage replication is similarly less useful if every copy depends on the same hardware or network path.
OpenStack availability zones, host aggregates, server groups, Kubernetes topology controls, and storage placement policies can all help distribute workloads. But logical placement does not automatically guarantee physical isolation. OpenStack explicitly notes that availability zones should not automatically be assumed to correspond to fault domains.
A useful architecture review therefore asks:
What single failure could make every copy of this service unavailable?
Treat GPU and Compute Failures as Normal Infrastructure Events
Expensive accelerated hardware should not become irreplaceable infrastructure.
A production architecture should expect a GPU or compute node eventually to leave service because of hardware problems, maintenance, driver issues, or other faults.
For training, that typically means ensuring the job can recover somewhere else without losing all completed work.
For inference, it usually means ensuring the disappearance of one node does not remove the only serving endpoint.
Infrastructure monitoring should also look beyond utilization. GPU health signals can include memory errors, PCIe issues, thermal and power conditions, drivers, and accelerator interconnects. NVIDIA's Data Center GPU Manager, for example, exposes health monitoring and diagnostics across those areas.
The scheduler may be able to replace a failed workload, but orchestration cannot recover state that was never saved.
That distinction becomes particularly important for long-running training.
Training Reliability Is Often About Recovery, Not High Availability
A long-running training job may not need uninterrupted operation, but it does need a reliable way to resume.
Consider a job that has already consumed three days of accelerator time.
If the node fails and the job restarts from a checkpoint written 20 minutes earlier, the incident may be manageable.
If there is no usable checkpoint, three days of compute may need to be repeated.
Production training environments therefore need a checkpoint strategy that answers:
- How frequently is state saved?
- How much progress can be lost?
- Where are checkpoints stored?
- Is checkpoint storage independent of the worker node?
- How long does restoration take?
- Has restoration actually been tested?
This last question matters.
A checkpoint is not a recovery plan until somebody has successfully restored from it.
Checkpoint frequency also creates an infrastructure trade-off. Writing checkpoints more often reduces potential lost work, but increases storage and network traffic. Recovery objectives therefore need to be considered alongside storage throughput and workload performance.
Inference Availability Depends on the Application Around It
Inference reliability should be designed according to what depends on the endpoint.
A single inference server may be perfectly reasonable during development. The same architecture becomes risky when a production application cannot function without it.
Business-critical inference services may require:
- multiple serving replicas;
- placement across appropriate failure domains;
- load balancing;
- application-aware health checks;
- automatic replacement of unhealthy instances;
- sufficient spare capacity after a replica fails;
- autoscaling where it is practical;
- sensible request timeouts and retry policies; and
- graceful application behavior when inference is unavailable.
Graceful failure is particularly important.
Sometimes the right response to an unavailable model is not unlimited retrying. The application might queue work, fall back to another workflow, return a limited response, or temporarily disable the dependent feature.
The infrastructure and application teams should decide that behavior before an outage occurs.
AI Storage Has More Than One Performance and Durability Requirement
Production AI does not create one generic requirement for “fast storage.” It creates several storage workloads with different characteristics.
For example:
Datasets may require substantial capacity and high sequential throughput.
Training checkpoints may generate large periodic writes and need fast restoration.
Model artifacts require durable storage and controlled versioning.
Embeddings and application state can introduce different latency and access patterns.
Logs and telemetry require retention for troubleshooting, security, and capacity analysis.
These requirements do not necessarily belong on the same storage tier.
Distributed storage platforms such as Ceph can place replicated or erasure-coded data according to defined failure domains. Ceph's CRUSH mechanism can model physical hierarchy such as hosts and racks so data placement accounts for correlated failures.
Infrastructure teams should also test storage during degraded operation.
A cluster rebuilding data after a disk or node failure consumes disk, CPU, and network resources. Storage that performs well under normal conditions may behave differently while simultaneously recovering and servicing AI workloads.
Networking Can Become Both a Performance and Reliability Constraint
A healthy GPU does not help much if it is waiting on the network.
Distributed training may create substantial east-west communication among workers. Compute nodes may continuously read datasets or write checkpoints to shared storage. Inference architectures add communication among applications, load balancers, serving instances, storage, and monitoring systems.
Teams should evaluate:
- east-west bandwidth;
- latency and packet loss;
- storage network connectivity;
- accelerator interconnect requirements;
- redundant network paths;
- contention between workload and recovery traffic; and
- workload placement relative to network topology.
This is why increasing GPU capacity alone does not necessarily increase usable AI capacity.
The bottleneck may move elsewhere.
Monitor the Service, Not Only the Accelerator
GPU utilization tells you whether the GPU is busy. It does not tell you whether the AI service is healthy.
Production observability should connect signals across the complete workload path.
Useful areas include:
The objective is correlation.
If inference latency rises, operators should be able to determine whether the cause is application behavior, insufficient serving capacity, GPU saturation, network degradation, or storage latency.
Google's Site Reliability Engineering guidance similarly emphasizes monitoring latency, traffic, errors, and saturation rather than treating individual infrastructure utilization metrics as complete measures of service health.
Capacity Planning Must Include Failure and Maintenance
Production capacity planning should account for unavailable capacity, not only normal demand.
A GPU cluster sized exactly for today's expected workload may have no room for a failed node, maintenance window, demand spike, or new deployment.
Planning should consider:
- peak versus average demand;
- expected workload growth;
- minimum capacity after a node failure;
- capacity temporarily unavailable during maintenance;
- competition between training and inference;
- storage and network growth alongside GPU growth; and
- how quickly additional accelerated hardware can actually become available.
Autoscaling does not eliminate the capacity problem if there is no physical accelerator capacity available to allocate.
For critical services, some headroom may therefore be part of the reliability design rather than wasted infrastructure.
Day 2 Operations Are Part of Production AI Architecture
If normal infrastructure maintenance causes an unacceptable service outage, the platform is not fully production-ready.
GPU drivers need updates. Operating systems need security patches. Kubernetes and OpenStack require lifecycle management. Storage clusters need upgrades. Hardware eventually needs repair or replacement.
Production environments need processes for:
- draining or evacuating workloads;
- replacing hardware;
- applying operating system and security updates;
- upgrading Kubernetes or OpenStack;
- updating GPU drivers;
- upgrading storage infrastructure;
- testing changes before broad rollout;
- rolling back failed changes; and
- validating workload health afterward.
Kubernetes, for example, distinguishes between involuntary failures and voluntary disruptions such as draining a node for repairs or upgrades, and provides disruption controls for replicated applications.
Maintenance should be treated as a routine infrastructure condition, not an unexpected event.
Who Owns the Incident When an AI Service Fails?
Technical resilience needs an equally clear operational ownership model.
An AI service may cross several teams:
- the model team owns model behavior;
- the application team owns the application consuming the model;
- the platform team owns orchestration;
- the infrastructure team owns compute, networking, and storage; and
- a cloud or infrastructure provider may operate part of the underlying environment.
Without predefined responsibility, incidents can become escalation loops.
Teams should know who receives alerts, who performs first-line triage, what each team is responsible for, when a provider becomes involved, and who owns recovery.
A runbook should answer those questions before an outage does.
Is Your AI Infrastructure Ready to Become Business-Critical?
Use this checklist as a practical production-readiness review.
Production AI Depends on More Than GPUs
Once AI workloads become operational dependencies, reliability depends on the infrastructure surrounding the accelerators.
Compute placement matters. Storage durability matters. Network topology matters. Kubernetes and OpenStack lifecycle management matters. So do monitoring, capacity planning, hardware replacement, and the people responsible for operating the platform.
These are infrastructure problems VEXXHOST already works with across OpenStack, Kubernetes, Ceph, networking, GPU infrastructure, and managed cloud operations. Current VEXXHOST infrastructure offerings span those underlying cloud and storage layers, including hosted and on-premises deployments.
The goal is not to make every AI workload highly available. It is to build infrastructure whose reliability matches how important that workload has become.
Moving AI into production therefore is not simply about adding more compute.
It is about building infrastructure that can fail, recover, scale, undergo maintenance, and continue to be operated predictably.
If your organization is evaluating the infrastructure behind production AI workloads, talk to VEXXHOST about the cloud infrastructure requirements behind your environment.
