GPUs serve as the fundamental computational engines for AI. However, In large-scale training environments, overall performance is not limited by processing speed, but by the speed of the network communication between them.
Large language models are trained on thousands of GPUs, which creates a huge amount of cross-GPU traffic. In these systems, even the smallest delays compound. A microsecond lag when GPUs share data can cause a chain reaction that adds hours to the training job. Therefore, these systems require a specialized network that’s designed to transfer large amounts of data with minimal delay.
The traditional approach of routing GPU data through the CPU created a severe bottleneck at scale. To fix this bottleneck, technologies like RDMA and GPUDirect were invented to essentially build a bypass around the CPU. This creates a direct path for GPUs to talk to one another.
This direct communication method needs a network that can handle the speed. The two main choices available today to provide this are InfiniBand and RoCEv2.
So, how do you choose between InfiniBand and RoCEv2? It’s a big deal, forcing you to balance raw speed against budget and how much hands-on tuning you’re willing to do.
Let’s take a closer look at each technology to see its strengths and weaknesses.
Basic Concepts
Before we compare InfiniBand and RoCEv2, let’s first understand how traditional communication works and introduce some basic concepts like RDMA and GPU Direct.
Traditional Communication
In traditional systems, most of the data movement between machines is handled by the CPU. When a GPU finishes its computation and needs to send data to a remote node, it follows the following steps –
- The GPU writes the data to the system (host) memory
- The CPU copies that data into a buffer used by the network card
- The NIC (Network Interface Card) sends the data over the network
- On the receiving node, the NIC delivers the data to the CPU
- The CPU writes it into system memory
- The GPU reads it from system memory
This approach works well for small systems, but it doesn’t scale for AI workloads. As more data gets copied around, the delays start to add up, and the network struggles to keep up.
RDMA
Remote Direct Memory Access (RDMA) enables a local machine to access the memory of a remote machine directly without involving the CPU in the data transfer process. In this architecture, the network interface card handles all memory operations independently, allowing it to read from or write to remote memory locations without creating intermediate copies of the data. This direct memory access capability eliminates the traditional bottlenecks associated with CPU-mediated data transfers and reduces overall system latency.
RDMA proves particularly valuable in AI training environments where thousands of GPUs must share gradient information efficiently. By bypassing operating system overhead and network delays, RDMA enables the high-throughput, low-latency communication essential for distributed machine learning operations.
GPUDirect RDMA
GPUDirect is NVIDIA’s way of letting GPUs talk directly to other hardware through PCIe connections. Normally, when a GPU needs to transfer data to another device, it has to take the long way around. The data goes from GPU memory to system memory first, then the receiving device grabs it from there. GPUDirect skips the CPU entirely. Data moves directly from one GPU to another.
GPUDirect RDMA extends this to network transfers by allowing the NIC to access GPU memory directly using PCIe.
Now that we understand concepts like RDMA and GPUDirect, let’s look into the infrastructure technologies InfiniBand and RoCEv2 that support GPUDirect RDMA.
InfiniBand
InfiniBand is a high-performance networking technology designed specifically for data centers and supercomputing environments. While Ethernet was built to handle general traffic, InfiniBand is designed to meet high speed and low latency for AI workloads.
It’s like a high-speed bullet train where both the train and the tracks are designed to maintain the speed. InfiniBand follows the same concept, everything including the cables, network cards, and switches are designed to move data fast and avoid any delays.
How does it work?
InfiniBand works completely differently from regular Ethernet. It doesn’t use the regular TCP/IP protocol. Instead, it relies on its own lightweight transport layers designed for speed and low latency.
At the core of InfiniBand is RDMA, which allows one server to directly access the memory of another without involving the CPU. InfiniBand supports RDMA in hardware, so the network card, called a Host Channel Adapter or HCA, handles data transfers directly without interrupting the operating system or creating extra copies of data.
InfiniBand also uses a lossless communication model. It avoids dropping packets even under heavy traffic by using credit-based flow control. The sender transmits data only when the receiver has enough buffer space available.
In large GPU clusters, InfiniBand switches move data between nodes with extremely low latency, often under one microsecond. And because the entire system is built for this purpose, everything from the hardware to the software works together to deliver consistent, high-throughput communication.
Let’s understand a simple GPU-to-GPU communication using the following diagram -
- GPU 1 hands off data to its HCA, skipping the CPU
- The HCA initiates an RDMA write to the remote GPU
- Data is transferred over the InfiniBand switch
- The receiving HCA writes the data directly to GPU 2’s memory
Strength
- Fast and predictable – InfiniBand delivers ultra-low latency and high bandwidth, keeping large GPU clusters running efficiently without hiccups.
- Built for RDMA – It handles RDMA in hardware and uses credit-based flow control to avoid packet drops, even under heavy load.
- Scalable – Since all parts of the system are designed to work together, performance is not impacted if additional nodes are added to the cluster.
Weaknesses
- Expensive – Hardware is expensive and mostly tied to NVIDIA, which limits flexibility.
- Harder to manage – Setup and tuning require specialized skills. It’s not as straightforward as Ethernet.
- Limited interoperability – It doesn’t play well with standard IP networks, making it less flexible for general-purpose environments.
RoceV2
RoCEv2 (RDMA over Converged Ethernet version 2) brings the benefits of RDMA to standard Ethernet networks. RoCEv2 takes a different approach than InfiniBand. Instead of needing custom network hardware, it just uses your regular IP network with UDP for transport.
Think of it like upgrading a regular highway with an express lane just for critical data. You don’t have to rebuild the entire road system. You just need to reserve the fast lane and tune the traffic signals. RoCEv2 uses the same concept, it delivers high-speed and low-latency communication using the existing Ethernet system.
How does it work?
RoCEv2 brings RDMA to standard Ethernet by running over UDP and IP. It works across regular Layer 3 networks without needing a dedicated fabric. It uses commodity switches and routers, making it more accessible and cost-effective.
Like InfiniBand, RoCEv2 enables direct memory access between machines. The key difference is that while InfiniBand handles flow control and congestion in a closed, tightly controlled environment, RoCEv2 relies on enhancements to Ethernet, such as –
Priority Flow Control(PFC) – Prevents packet loss by pausing traffic at the Ethernet layer based on priority.
Explicit Congestion Notification(ECN) – Mark packets instead of dropping them when congestion is detected.
Data Center Quantized Congestion Notification(DCQCN) – A congestion control protocol that reacts to ECN signals to manage traffic more smoothly.
To make RoCEv2 work well, the underlying Ethernet network needs to be lossless or close to it. Otherwise, RDMA performance drops. This requires careful configuration of switches, queues, and flow control mechanisms throughout the data center.
Let’s understand a simple GPU-to-GPU communication using the following diagram with RoCEv2 –
- GPU 1 hands off data to its NIC, skipping the CPU.
- The NIC wraps the RDMA write in UDP/IP and sends it over Ethernet.
- Data flows through standard Ethernet switches configured with PFC and ECN.
- The receiving NIC writes the data directly to GPU 2’s memory.
Strength
Cost-effective – RoCEv2 runs on standard Ethernet hardware, so you don’t need a specialized network fabric or vendor-locked components.
Easier to deploy – Since it uses familiar IP-based networking, it’s easier for teams already managing Ethernet data centers to adopt.
Flexible integration – RoCEv2 works well in mixed environments and integrates easily with existing Layer 3 networks.
Weaknesses
Requires tuning – To avoid packet loss, RoCEv2 depends on careful configuration of PFC, ECN, and congestion control. Poor tuning can hurt performance.
Less deterministic – Unlike InfiniBand’s tightly controlled environment, Ethernet-based networks can introduce variability in latency and jitter.
Complex at scale – As clusters grow, maintaining a lossless Ethernet fabric with consistent behavior becomes increasingly difficult.
Conclusion
In a large-scale GPU cluster, compute power is worthless if the network can’t handle the load. Network performance becomes just as vital as the GPUs because it holds the whole system together. Technologies like RDMA and GPUDirect RDMA help cut out the usual slowdowns by getting rid of unnecessary interruptions and CPU copying, letting GPUs talk directly to each other.
Both InfiniBand and RoCEv2 speed up GPU-to-GPU communication, but they take different approaches. InfiniBand builds its own dedicated network setup. It provides excellent speed and low latency, but at a very high cost. RoCEv2 provides more flexibility by using the existing Ethernet setup. It’s easier on the budget, but it needs proper tuning of PFC and ECN to make it work.
At the end of the day, it’s a classic trade-off. Go with InfiniBand if your top priority is getting the absolute best performance possible, and budget is less of a concern. But if you want a more flexible solution that works with your existing network gear and costs less upfront, RoCEv2 is the way to go.
