GPUs and TPUs are both machines for multiplying large matrices quickly, and nearly everything in modern AI reduces to that operation. The useful question for a buyer, a developer or a reader of a cloud pricing page is not "which is faster" but "which constraint binds my workload, and are these two spec sheets even measuring the same thing". This guide compares NVIDIA's H100 with Google's TPU v5p and v6e from the vendors' own documentation, then shows with one simple calculation why memory bandwidth, more than peak FLOPS, often decides real speed. For the serving techniques that follow from this, see Inference Serving Explained.
How the two designs are organised
A GPU as NVIDIA describes it consists of "a number of Streaming Multiprocessors (SMs), on-chip L2 cache, and high-bandwidth DRAM."[3] Work is organised as threads grouped into thread blocks, and each block runs on one SM, so a launch needs many blocks to fill the chip; the guide says to aim for several times more blocks than SMs.[3] Matrix work is done by Tensor Cores, introduced in the Volta architecture to accelerate matrix multiply-and-accumulate operations.[3] The Hopper post gives the H100 SXM5 configuration as 132 SMs and 528 fourth-generation Tensor Cores, four per SM.[2] It also describes a Transformer Engine that "intelligently manages and dynamically chooses between FP8 and 16-bit calculations."[2]
A TPU is organised differently. Google's documentation says each TensorCore "consists of one or more matrix-multiply units (MXUs), a vector unit, and a scalar unit."[4] The MXU is a systolic array: "thousands of multiply-accumulators that are directly connected to each other to form a large physical matrix," so data moves between neighbouring units during a multiply without extra memory accesses.[4] The vector unit handles general computation such as activations and softmax, and the scalar unit handles control flow and address calculation.[4] On v6e and TPU7x each MXU is 256 x 256 and performs 128K bfloat16 multiply-accumulates per cycle; v5p and earlier use 128 x 128 units.[4] Newer TPUs also include SparseCores for sparse operations such as the embedding lookups in recommendation models.[4]
The original TPU paper (2017) frames the design goal as inference for production workloads, centred on a large 8-bit matrix unit and a deterministic execution model, and reports 15 to 30 times the speed of the contemporary CPU and GPU it tested, on Google's own workloads.[7] That is history and a vendor-authored comparison against 2015-era hardware, not evidence about today's chips.
Per-chip specifications, with the caveats
| Spec | NVIDIA H100 SXM | Google TPU v5p | Google TPU v6e |
|---|---|---|---|
| Memory per chip | 80 GB | 95 GiB usable (96 GiB physical) | 32 GB (31.25 GiB usable) |
| Memory bandwidth | 3.35 TB/s | 2,765 GBps | 1,638 GBps |
| Peak BF16 compute | 1,979 TFLOPS, listed "with sparsity" | 459 TFLOPs | 918 TFLOPs |
| Chip-to-chip link | NVLink, 900 GB/s | ICI, 1,200 GBps bidirectional per chip | ICI, 800 GBps bidirectional per chip |
| Largest documented unit | not stated on the product page | 8,960 chips per pod | 256-chip pod |
Sources: NVIDIA's H100 page, Google Cloud's v5p and v6e pages, all read on 2026-10-27.[1][6][5] H100 SXM lists the figures above; the NVL variant differs.[1]
Four caveats stop this table from being read as a scoreboard:
- Sparsity. NVIDIA's asterisk on its Tensor Core rows reads "With sparsity."[1] The Hopper post lists the dense BF16 peak as 1000 TFLOPS against 2000 with sparsity, and says its figures are preliminary.[2] Our reading: a dense workload should use the lower figure, roughly half the page's headline. The TPU figures are not labelled sparse.
- Units. GB, GiB, GBps and TB/s appear side by side. A GiB is about 7% larger than a GB. Google also separates physical from usable memory.[6][5]
- Interconnect. NVIDIA's NVLink figure is described as 900 GB/s total bandwidth,[2] while Google's ICI figures are "bidirectional" per chip.[6][5] Do not assume the two numbers mean the same thing.
- Topology. What matters at scale is how chips are wired together. A TPU slice uses a 2D or 3D layout, and cubes of 64 chips are connected by copper links inside and optical circuit switches between cubes.[4][6] The TPU v4 paper describes that optically reconfigurable design and reports a 4,096-chip system compared with other vendors' systems, again as the authors' own comparison.[8] The H100 page documents no equivalent unit, so a fair system-level comparison needs the specific cluster configuration on offer.
The calculation that matters more: operations per byte
NVIDIA's performance guide gives the framework. A function's arithmetic intensity is its operations divided by the bytes it accesses; a processor's ops:byte ratio is its math bandwidth divided by its memory bandwidth; and "an algorithm is memory limited if its arithmetic intensity is lower than the processor's ops:byte ratio."[3] Runtime is roughly the longer of memory time and math time.[3] The guide's own example, for a V100 in FP16, is a linear layer at batch size 512 with about 315 FLOPS per byte (math limited) against about 1 FLOPS per byte at batch size 1 (memory limited).[3]
Applying that to the table is our arithmetic, using peak figures that real code does not reach:
| Chip | Peak BF16 (dense) | Memory bandwidth | Ops per byte (peak / bandwidth) |
|---|---|---|---|
| H100 SXM | about 1,000 TFLOPS (Hopper post, preliminary) | 3.35 TB/s | about 300 |
| TPU v5p | 459 TFLOPs | 2.765 TB/s | about 166 |
| TPU v6e | 918 TFLOPs | 1.638 TB/s | about 560 |
Now consider generating one token for one user. Each 2-byte BF16 weight is read once and used for one multiply-add, which is two operations, so intensity is about one operation per byte. That is hundreds of times below every ratio in the table, so the chip waits on memory and most of its arithmetic units idle. Batching helps because one weight read then serves many users: in this simplified model intensity rises roughly in step with the batch, so a batch of a few hundred tokens is where compute starts to bind. The simplification ignores the KV cache, activations and attention, all of which add memory traffic (see Qwen3.6-27B: Hybrid Attention and KV-Cache Memory Math), so treat the break-even batch as a rough order of magnitude.
Three practical readings follow, which are our interpretation rather than vendor claims:
- For low-latency, small-batch serving, compare memory bandwidth per dollar before peak FLOPS. On these specifications the v6e chip has a high peak compute to bandwidth ratio, which means it needs a large batch to be used efficiently.
- For training and large-batch work, peak compute and interconnect both matter, because the matrices are large and chips must exchange results every step.
- Capacity limits are separate from speed limits. A model that does not fit in a chip's memory must be split across chips, and then the interconnect enters the calculation.
What this article does not tell you
It does not rank the chips on cost, throughput or energy; no benchmark was run, and the vendor figures are peaks. Pricing varies by provider, region and commitment, and the H100 and the two TPUs are different product generations. Software maturity, model support and your team's experience often outweigh a spec difference of this size. When a provider or vendor claims a speed-up, ask which precision, whether the figure is sparse, what batch size and which model, and compare with How to Read an AI Benchmark Claim Without Being Misled. For the cost side of the same constraint, see Tokens, Context Windows and What an AI Request Really Costs.
Sources and method
All specifications were read from the vendors' pages on 2026-10-27 and may have changed since. Where a figure is derived (ops per byte, the batch estimate), the calculation is shown above. The NVIDIA Hopper post dates from March 2022 and its throughput table is labelled preliminary, so prefer the current product page for anything contractual.
