A model with hundreds of billions of parameters does not fit on one chip, and even when it does, training it on one chip would take far too long. So the work is spread over thousands of accelerators that must constantly exchange results, and the network between them becomes part of the computer. This guide explains AI cluster networking from the vendors' and operators' own documents: why there are two networks, how much slower each step outward is, what Meta learned running Ethernet for AI training, and what the new Ultra Ethernet specification is trying to fix. It builds on GPU vs TPU for AI, which covers a single chip, and Inference Serving Explained.
Two networks, two jobs
NVIDIA's NVLink page calls NVLink and its switch a "scale-up networking fabric with high-bandwidth GPU-to-GPU communications for AI training, inference, and other demanding rack-scale GPU-accelerated workloads."[1] The aim is stated plainly: all-to-all connectivity so that a rack can act as one device. On the same page, the Vera Rubin NVL72 rack connects 72 GPUs "in an all-to-all topology for a total of 216 TB/s," and NVIDIA says those 72 GPUs "can be used as a single high-performance accelerator."[1] The page labels its figures "Preliminary specifications; may be subject to change."[1]
Beyond the rack is the scale-out network. Meta's engineering post describes the split in its own clusters: a backend network that "connects all RDMA NICs in a non-blocking architecture" for training, kept separate from a frontend network used for "data ingestion, checkpointing, and logging."[4] Meta says it built the backend as a dedicated network so it could "evolve, operate, and scale independently from the rest of the data center network."[4]
That gives the vocabulary used in the rest of this article:
- Scale-up: links among the accelerators of one server or rack. Highest bandwidth, shortest distances, usually vendor-specific.
- Scale-out: links between racks across a data centre. Lower bandwidth per chip, longer distances, and increasingly built on Ethernet.
- Frontend network: storage, data loading and management traffic, kept away from training traffic.
The bandwidth ladder
Google's TPU v5p documentation is useful because one table puts three levels side by side for the same chip.[3]
| Link (TPU v5p, per chip) | Documented figure | In GB/s (our conversion) |
|---|---|---|
| HBM memory bandwidth | 2,765 GBps | 2,765 |
| Inter-chip interconnect (ICI), bidirectional | 1,200 GBps | 1,200 |
| Data centre network (DCN) | 50 Gbps | 6.25 |
Source: Google Cloud TPU v5p page, read 2026-11-06.[3] The documentation also lists the ICI topology as a 3D torus and says a v5p pod has 8,960 chips.[3] The conversion in the last column divides gigabits by eight and ignores protocol overhead.
Our reading: moving from memory to the inter-chip link costs a factor of about two, and moving from the inter-chip link to the data centre network costs a factor of roughly 190. If each link ran at its documented figure, a gigabyte would take about 0.4 ms to stream from memory, 0.8 ms over ICI and 160 ms over the data centre network. Treating the bidirectional ICI figure as a one-way rate flatters ICI, so its time is a lower bound. These are idealised times that ignore latency and sharing, but the shape is the point: each step outward is slower, and the last step is the steepest.
NVIDIA's pages show the same pattern for GPUs. The NVLink page lists per-GPU NVLink bandwidth of 900 GB/s for the fourth generation (Hopper), 1,800 GB/s for the fifth (Blackwell) and 3,000 GB/s for the sixth (Vera Rubin), and says the sixth generation is "12x the bandwidth of PCIe Gen6."[1] The 2022 Hopper post made the same comparison for the fourth generation: 900 GB/s total, "7x the bandwidth of PCIe Gen 5."[2] The page's aggregate column is consistent with per-GPU speed multiplied by GPU count: 8 GPUs at 900 GB/s give the listed 7.2 TB/s, and 72 GPUs at 3,000 GB/s give 216 TB/s. That multiplication is ours, not NVIDIA's statement of method.
Why training software cares: keep the chatty traffic close
The Megatron-LM paper explains how this hierarchy shapes training. Tensor parallelism splits each layer across GPUs, and it requires an all-reduce at every layer. The authors write that for larger models "the all-reduce communication required for tensor parallelism needs to go through inter-server links, which are slower than the high-bandwidth NVLink available within a multi-GPU server."[7] Their guidance, labelled Takeaway #1: "tensor model parallelism should generally be used up to degree g when using g-GPU servers, and then pipeline model parallelism can be used to scale up to larger models across servers."[7]
The paper's headline result shows what that buys: a model with 1 trillion parameters trained at 502 petaFLOP/s on 3,072 GPUs, 52% of theoretical peak per GPU.[7] The authors add that "at scale, distributed training is communication-intensive" and report an effective bisection bandwidth of 892 GB/s for pipeline-parallel communication and 13 TB/s for data-parallel communication on that run.[7] They also say that slower inter-node interconnects or more communication-heavy partitionings "would hinder scaling performance."[7] The study is from 2021 on A100-generation hardware, so the specific numbers are dated; the logic of matching communication-heavy parallelism to fast links is the lasting part.
What Meta found running Ethernet for AI training
Meta's 2024 post is a rare operator's account. It says Meta chose "RDMA over Converged Ethernet version 2 (RoCEv2) as the inter-node communication transport for the majority of our AI capacity."[4] The backend uses a two-stage Clos topology it calls an AI Zone: rack switches provide connectivity within the rack over copper cables, and spine switches connect racks over fibre with 400G pluggable transceivers. Because training now exceeds one zone, an aggregator layer links zones, and "the cross-AI Zone connectivity is oversubscribed by design," so Meta changed its job scheduler to place training nodes so as to reduce cross-zone traffic.[4]
The routing story is the most instructive part:
- AI traffic is unlike web traffic. Meta lists three properties: low entropy (few, repetitive flows), burstiness, and "elephant flows" that can reach NIC line rate.[4]
- Standard ECMP performed poorly. ECMP spreads flows by hashing packet headers, which works when there are many diverse flows. Meta writes that "ECMP rendered poor performance for the training workload due to the low flow entropy."[4]
- Pinning paths worked until jobs were fragmented. A path-pinning scheme did fine for fully allocated racks, but partial allocation degraded training performance "up to more than 30%," and Meta temporarily doubled uplink bandwidth, which "required 2x network capacity."[4]
- More flows helped. Queue-pair scaling with an enhanced ECMP hash improved the AllReduce collective by "up to 40%" compared with baseline ECMP, though Meta calls the probabilistic hashing a persistent downside.[4]
- Congestion control was reworked. At 400G, Meta says tuning the standard DCQCN scheme degraded performance, and it ran "over a year of experience with just PFC for flow control, without any other transport-level congestion control," then co-designed the collective library and transport for receiver-driven admission.[4]
These are one company's results on its own workloads and network. They support a narrow claim, that default data-centre routing assumptions break on AI training traffic, and not a general ranking of Ethernet against other fabrics.
What Ultra Ethernet is trying to change
The Ultra Ethernet Consortium released its 1.0 specification on 11 June 2025, describing it as "an Ethernet-based communication stack" for AI and HPC covering NICs, switches, optics and cables.[5] Its 2024 technical preview lists what the new transport (UET) does, and several items map directly onto Meta's problems:[6]
- Packet spraying. "UET senders spray packets across many paths to the destination, avoiding the ECMP flow collision problem," shifting traffic away from a path judged congested.[6]
- Works without a lossless network. UET "is designed to operate well over best-effort networks that may drop packets," while also performing well where Priority Flow Control is used.[6]
- No connection handshake. An "ephemeral connection approach" removes connection set-up latency and reduces the state a NIC must hold.[6]
- Profiles. Three profiles (AI Base, AI Full and HPC) select subsets of the specification.[6]
The consortium's site menu now lists a 1.1 specification download; this article did not open it, so it makes no claim about what 1.1 changed. It also does not assess which vendors ship compliant hardware. The press release describes "active implementations and compliance programs already in progress,"[5] which is the consortium's statement, not independent verification.
How to use this when reading a cluster announcement
- Ask which network a number belongs to. "Terabytes per second" of GPU bandwidth usually refers to a scale-up domain such as 72 GPUs, not to the whole data centre.
- Ask how big the fast domain is. Tensor-parallel work stays inside it, so a bigger fast domain widens the choices open to model designers. The sources document 8 and 72 GPUs for NVLink domains, and 8,960 chips per TPU v5p pod.[1][3]
- Look for oversubscription. Meta's cross-zone links are oversubscribed on purpose; a headline cluster size says nothing about bandwidth between its far ends.
- Treat open-standard claims as claims. A specification is not a deployment. Look for named operators, measured results and independent testing.
What this article does not cover
It does not compare NVLink, InfiniBand and Ethernet on cost, latency or real throughput; no benchmark was run, and the sources do not provide a like-for-like one. InfiniBand appears only in the cited papers as the inter-server link in the Megatron-LM experiments and is not described further here. Optical switching, in-network reduction and photonic links are outside scope. All vendor pages were read on 2026-11-06 and can change; the NVIDIA Hopper post dates from 2022 and its figures are described there as preliminary.
