Inside an AI Data Center: How Switches Connect Thousands of GPUs
When people picture an AI data center, they picture the chips: rows of GPUs, blinking quietly, crunching the math behind the models we all now use. What they rarely picture is the part that actually decides whether all those chips earn their keep: the network that ties them together.
Here is the uncomfortable truth that anyone who has built one of these clusters will tell you. A single GPU is fast. A thousand GPUs are not automatically a thousand times faster. Whether they come anywhere close depends almost entirely on how well they can talk to each other, and that is a networking problem, not a silicon one.
This is the story of the switches, cables, and protocols that turn a warehouse full of expensive chips into a single machine. It is also the story of how, over the past two years, the industry quietly changed its mind about how to wire them.
Why a Model Needs the GPUs to Talk
Training a large model isn't one big calculation. It is billions of small ones, spread across every GPU in the cluster. Each chip works on its own slice of the data, computes how the model should change, and then (this is the key part) has to share what it learned with every other chip before the next step can begin.
That sharing step is relentless. Picture thousands of GPUs all finishing a chunk of work at roughly the same instant and needing to exchange their results at once. The technical name for this pattern is a collective operation (the most common one is an "all-reduce," where every GPU's contribution is summed and the result handed back to everyone). What matters for our purposes is the shape of it: a sudden, synchronized, all-at-once burst of traffic, repeated millions of times over a training run.
Two consequences fall out of this, and they explain almost everything about how AI networks are designed:
- First, the network is the metronome. The whole cluster cannot start the next step until the slowest exchange finishes. One congested link, one delayed packet, and thousands of GPUs sit idle waiting. At the prices these chips command, idle time is the most expensive thing in the building.
- Second, ordinary networking assumptions break. The internet is built to tolerate the occasional lost packet; your video call drops a frame and recovers. An AI fabric cannot afford that. A dropped packet means a retransmission, a retransmission means delay, and delay means the metronome skips. These networks must be effectively lossless, which is a demanding thing to ask of any network.
Three Rings of Connection
The trick to wiring thousands of GPUs is that you do not wire them all the same way. Modern AI data centers think in three concentric rings, each with its own technology.
!Three concentric rings showing scale-up at the center, scale-out in the middle, and scale-across on the outside
- Scale-up is the innermost ring: the GPUs inside a single server or rack, connected so tightly they almost behave like one giant chip. This is the domain of dedicated, ultra-fast links (such as NVIDIA's NVLink) that let a handful of GPUs share memory at staggering bandwidth. Scale-up is short-range and brutally fast.
- Scale-out is the middle ring, and it is where the switches we are really talking about live. This is how you connect rack to rack, server to server, until you have stitched together thousands or tens of thousands of GPUs into one cluster. Scale-out is a network in the traditional sense, with switches, optical cables, and addresses, just an unusually demanding one.
- Scale-across is the outermost ring, and the newest. As single buildings run out of power and space, operators are starting to link multiple data centers into one logical cluster. NVIDIA and Broadcom both introduced this concept of stretching GPU connectivity across sites, extending the fabric over much longer distances.
The art of cluster design is deciding what goes in which ring, because each ring trades cost against performance differently.
The Topology: How You Arrange the Switches
Within the scale-out ring, you cannot simply plug every GPU into every other GPU; the cabling would be impossible. Instead, networks are built in tiers, most commonly a "leaf-spine" arrangement.
Think of it like an airport hub system. Each rack of GPUs connects to a "leaf" switch sitting at the top of that rack. The leaf switches do not connect to each other directly; instead, they all connect upward to a layer of "spine" switches. Any GPU can reach any other GPU by going up to a spine and back down, remaining never more than a couple of hops away. Add more spines and you add more parallel paths, which means more capacity for those all-at-once traffic bursts.
!Leaf-spine topology diagram: three spine switches at the top, four leaf switches below them fully meshed, each leaf connected down to a rack of GPUs
The clever refinements get more specialized from there. AI clusters often use "rail-optimized" designs, where corresponding GPUs across many racks share a dedicated path, precisely because the traffic patterns of collective operations are so predictable. When you know in advance that GPU #3 in every rack will need to talk to GPU #3 in every other rack, you can build the network to favor exactly that.
The Fight Over the Fabric: InfiniBand vs. Ethernet
For roughly a decade, the answer to what to build this with was simple: InfiniBand. It is a networking technology purpose-built for high-performance computing, prized for two things: extremely low latency (around one to two microseconds) and a design that essentially never drops packets. It became the default for serious GPU clusters, helped along by the fact that NVIDIA, which sells most of the GPUs, also owns the dominant InfiniBand product line through its Mellanox acquisition.
The catch was cost and lock-in. InfiniBand hardware has run roughly three times the price of comparable Ethernet switches, and it ties you to essentially one vendor's ecosystem.
Ethernet (the same basic technology that runs the rest of the internet and nearly every corporate network) was long dismissed as "good enough but not great" for AI. It was cheaper and familiar, but it dropped packets and added latency, the two things an AI fabric can least tolerate.
That gap has now largely closed. A set of refinements, including lossless flow-control mechanisms, congestion management, and smarter load balancing, let modern Ethernet behave much more like InfiniBand. The decisive moment came when the Ultra Ethernet Consortium released its UEC 1.0 specification, reworking the Ethernet stack specifically for the AI and HPC era.
The market responded fast: Ethernet overtook InfiniBand in AI back-end networking, a striking reversal, given that InfiniBand held close to 80% of that market just two years earlier. By the first quarter of 2026, Ethernet accounted for roughly two-thirds of switch sales in AI clusters. Even NVIDIA, InfiniBand's champion, now sells an AI-tuned Ethernet platform (Spectrum-X) alongside it, and that line has been one of its fastest-growing products.
Notably, this is not a clean knockout. InfiniBand still rebounded in absolute sales and remains the gold standard for the largest, most tightly-coupled training runs, where its consistency and predictability are worth paying for. Independent testing tends to land on a nuanced verdict: well-tuned Ethernet reaches parity with InfiniBand for the majority of workloads, and the old "Ethernet is far slower" reputation mostly reflects poorly configured networks rather than a hard ceiling. The right choice turns out to be workload-specific, not religious.
Closing Thoughts
Step back and the shift makes sense. The reason Ethernet is winning isn't that it suddenly became faster than InfiniBand. It is that, once the performance gap shrank to something tolerable, everything else tilted the decision: lower cost, a multi-vendor supply chain you are not hostage to, and the simple fact that the engineers you already employ know how to run it. On a cluster costing tens or hundreds of millions of dollars, the money saved on the network buys more GPUs, and more GPUs is the whole point.
It is a familiar pattern in computing history. The specialized, superior technology opens the frontier; the cheaper, open, "good enough" one industrializes it. Ethernet has played that role before, and it is playing it again.
So the next time you read about a new model trained on some staggering number of GPUs, remember that the number itself is only half the story. The other half is the unglamorous lattice of switches and cables that decided whether those chips spent their time computing or waiting. In an AI data center, the network isn't the plumbing behind the real work. Increasingly, **it *is* the work**.