Learn how AI workloads change network design across GPU clusters, storage, east-west traffic, RDMA/RoCE, Kubernetes placement, and production inference.
Powerful GPUs get most of the attention in AI infrastructure planning. But once AI workloads extend beyond a single server, accelerator performance depends on how quickly compute nodes, storage systems, and applications communicate.
A deployment can have enough GPU capacity and fast storage and still underperform because the network was designed for a different traffic pattern. Distributed training can generate sustained communication among workers. Data pipelines move datasets toward compute. Checkpoints create large periodic transfers. Production inference adds latency-sensitive request paths.
AI infrastructure is not simply conventional cloud infrastructure with GPUs added. At scale, the network becomes part of the compute system.
Why GPUs Change the Networking Conversation
The first networking question should not be “How fast is the NIC?” It should be: What needs to communicate, how often, and across which paths?
Depending on the workload, GPUs may exchange data with other GPUs, shared storage, orchestration services, or model-serving components. Not all GPU-to-GPU communication crosses the data-center network: communication inside a server may use PCIe or dedicated accelerator interconnects. But once a distributed workload spans nodes, network topology and performance become part of the execution path.
NVIDIA’s NCCL, for example, provides topology-aware collective operations such as AllReduce, AllGather, ReduceScatter, and AlltoAll for multi-GPU communication across processes and machines.
For the underlying communication model, see the NVIDIA NCCL documentation.
As GPU clusters grow, network design increasingly determines whether additional compute can work efficiently together.
AI Traffic Is Not One Type of Traffic
Different AI workloads create different network demands.
Distributed training often generates heavy east-west traffic as workers exchange data and synchronize progress. A slow or congested path can affect multiple workers because collective operations depend on coordinated communication.
Data ingestion and preprocessing create another pattern. Large datasets may be read in parallel from object, block, or file storage, potentially competing with other traffic.
Inference tends to be more request-driven. Real-time serving may prioritize latency and consistency, while batch inference can emphasize throughput. Multi-node inference can also create significant internal traffic depending on the serving architecture.
Checkpointing and model movement add periodic bursts as training state and model artifacts are written or moved.
Several of these patterns may occur at once. Designing only for average utilization can miss the periods that matter most.
For a broader comparison of the infrastructure differences between the two workload types, see AI Workloads on Kubernetes: Training vs. Inference Infrastructure Requirements.
East-West Traffic Becomes More Important
North-south traffic describes data entering or leaving an environment. East-west traffic describes communication among systems inside it.
Traditional application environments often emphasize north-south traffic. Distributed AI can shift much more activity inside the infrastructure itself, putting pressure on switching capacity, rack-to-rack links, topology, and oversubscription.
Oversubscription is especially important. A server may have a high-speed interface, but that does not mean every server can use its full bandwidth simultaneously. If multiple GPU nodes converge on a shared uplink, the aggregate path may become the real limit.
This does not mean every AI cluster requires a non-oversubscribed fabric. Teams need to know where oversubscription exists, which workloads cross those boundaries, and what happens as the cluster grows.
Bandwidth Matters, but It Is Not the Only Metric
A high link speed is useful, but it does not guarantee good AI performance.
Bandwidth determines how much data a path can carry. Latency affects how quickly communication completes. Packet loss and congestion can reduce effective throughput. Latency variation can make request performance less predictable. Oversubscription determines whether expected bandwidth remains available when many systems communicate simultaneously.
Consistency under load therefore matters as much as interface speed.
A representative test is more useful than an isolated benchmark. Testing storage from one host does not show what happens when several GPU workers read data while another job checkpoints and an inference service handles traffic.
When RDMA and RoCE Matter
Remote Direct Memory Access, or RDMA, allows compatible systems to transfer data directly between application memory with less CPU and kernel involvement in the data path. RoCE, or RDMA over Converged Ethernet, provides RDMA over Ethernet.
That can be valuable for communication-intensive distributed workloads where reducing communication overhead and latency materially improves performance. But RDMA and RoCE should be workload-driven design choices, not universal AI requirements.
Single-node training, many inference deployments, and smaller clusters may perform well on conventional Ethernet. Larger environments should still validate whether the application, communication libraries, adapters, and network fabric can benefit from RDMA before accepting the added operational complexity.
RoCE also raises fabric-design considerations. NVIDIA documentation describes mechanisms including ECN and PFC in RoCE environments, reinforcing that high-performance transport depends on congestion management and configuration, not just enabling a protocol.
For more detail, see NVIDIA’s RoCE networking documentation.
The Storage Network Can Become the GPU Bottleneck
A fast GPU cannot process data it has not received.
Training environments can generate parallel dataset reads while simultaneously writing checkpoints or moving model artifacts. If storage and compute traffic share constrained paths, they can compete for capacity.
Ceph is a useful example. Ceph clients communicate directly with OSDs, while OSD replication and recovery create additional network traffic. Ceph can use a single public network or separate client-facing and cluster networks depending on the deployment and traffic profile.
The broader lesson is simple: storage throughput measured at the storage layer is not necessarily the throughput available to GPU nodes. The entire path between data and compute has to support the expected concurrency.
The Ceph network configuration reference provides more detail on these traffic paths. VEXXHOST also provides managed Ceph storage for block, object, and file workloads, including AI and ML data pipelines.
Workload Placement Is Also a Networking Decision
Scheduling a workload onto an available GPU is not always the same as placing it well.
If distributed workers sit behind different oversubscribed links or far from the storage paths they depend on, the job may perform differently from one with better locality.
Kubernetes can expose GPUs as schedulable resources through device plugins. Node affinity, pod affinity, topology labels, and topology spread constraints can also help express placement and failure-domain requirements.
See the official Kubernetes guidance on GPU scheduling and topology spread constraints.
But Kubernetes cannot create physical bandwidth that does not exist. Scheduler policy is useful only when infrastructure topology is understood and exposed meaningfully.
GPU availability, network locality, storage proximity, and failure domains may all need to influence placement.
Training and Inference Should Not Automatically Get the Same Network Design
These are tendencies, not rules. Distributed inference may generate significant internal traffic, while training contained within one server may place little load on the data-center network.
The network should follow the workload architecture, not simply whether it is called “training” or “inference.”
Signs Your AI Network May Be Becoming the Constraint
Possible indicators include GPUs spending unexpected time waiting, multi-node jobs scaling poorly compared with single-node runs, training data arriving more slowly than storage benchmarks suggest, network utilization spiking during checkpoints, packet loss or congestion under distributed load, or inference latency becoming inconsistent when other infrastructure activity increases.
None of these symptoms proves networking is the cause. Storage latency, CPU contention, GPU topology, application design, and scheduling can produce similar effects.
Diagnosis should correlate GPU, compute, storage, network, and application metrics rather than treating each layer independently.
Questions to Ask Before Designing an AI Network
Before choosing a network architecture, ask:
- Are we supporting training, inference, or both?
- How many GPUs need to communicate simultaneously?
- Which communication stays within a server, and which crosses nodes?
- What east-west traffic will distributed workloads generate?
- Where do training data, checkpoints, and model artifacts live?
- How quickly must storage deliver data to GPU nodes?
- Will training, inference, and storage traffic share network capacity?
- Where does oversubscription exist?
- What changes when the GPU environment doubles in size?
- Are workload placement decisions topology-aware?
- Does the workload actually benefit from RDMA or RoCE?
- How will bandwidth, latency, loss, congestion, and application performance be observed together?
The objective is not to choose the fastest network technology available. It is to design a network appropriate for the workload, scale, and operating model.
Design AI Infrastructure as a Connected System
GPU, compute, networking, storage, and orchestration cannot be sized independently.
A faster GPU can expose a network bottleneck. A faster network can expose storage limitations. Adding nodes changes east-west traffic. Moving workers across racks changes communication paths. Storage recovery can consume capacity that training or inference expects to use.
At the virtual network layer, OpenStack Neutron with OVN supports distributed east-west routing between compute nodes. This does not replace physical network design, but it demonstrates why the physical and software-defined networking layers need to be considered together.
VEXXHOST’s GPU Infrastructure offering follows a workload-first approach: compute, storage, networking, platform, and operating responsibility are designed around the workload, with hosted or customer-premises deployment options.
Where appropriate, that infrastructure can connect with VEXXHOST’s OpenStack platform, Kubernetes services, and Ceph storage to create a broader open infrastructure environment.
Planning GPU capacity without planning how GPUs, storage, compute, and applications communicate can leave expensive AI infrastructure waiting on the network.
If your organization is planning or scaling AI infrastructure, talk to VEXXHOST about your workload, topology, storage, networking, and operating requirements before selecting the architecture.
