1.84×

the H200's KV cache on an MI300X with 70 GB of weights loaded, from 1.36× the memory

Hardware, July 2026

MI300X against H200.

One AMD Instinct MI300X against one NVIDIA H200, inference and training, on the same vLLM and PyTorch versions.

What we found

There is no single winner
The result splits by whether the model plus its KV cache fits in 141 GB.
The H200 wins from the lower paper specification
The MI300X leads on paper on every axis. Measured, the H200 sustains 1.25× the BF16 GEMM, 1.22× the FP8 GEMM and 1.11× the memory bandwidth. The cause is achieved efficiency.
Memory turns it around at 70B
Both hold 70 GB of FP8 weights. Weights are a fixed subtraction, so a 1.36× memory advantage becomes a 1.84× KV cache advantage. Beyond roughly 11 concurrent full-length requests the H200 is oversubscribed: requests queue and preempt, throughput falls and time to first token rises.
The largest model does not run on the H200
Llama 3.3 70B in BF16 is 141 GB of weights. The weights fit on the H200. Allocating KV cache afterwards does not.
Neither GPU throttles
Over 30 minutes of continuous load, 8B at concurrency 64, the MI300X held 99.2% of its peak and the H200 99.7%.

The results

Who won what

Neither wins outright. The H200 leads on smaller work and at low load; the MI300X takes over when the 70B model and its cache crowd the H200's memory.

MI300X leadsH200 leads

Mixture-of-experts decode, concurrency 1
2.72×
Dense 8B, prefill-heavy, concurrency 256
2.06×
Dense 8B, balanced, concurrency 256
1.40×
Dense 8B, single stream
Tie, 1.02×
Dense 70B FP8, single stream
1.12×
Dense 70B FP8, balanced, concurrency 128
1.14×
Dense 70B FP8, decode-heavy, concurrency 256
1.08×
Dense 70B BF16 (141 GB weights)
H200 cannot load it
only one
Training, 8B, BF16
1.32× to 1.40×
1.40×

On paper, and measured

The MI300X leads every line of the data sheet. Measured, the lead flips to the H200 on all three that were tested.

MI300X leadsH200 leads

On paper

BF16 dense
1,307.4 TFLOP/s against 989.4 TFLOP/s
1.32×
FP8 dense
2,614.9 TFLOP/s against 1,978.9 TFLOP/s
1.32×
HBM bandwidth
5.3 TB/s against 4.8 TB/s
1.10×
Capacity
192 GB against 141 GB
1.36×

Measured, best sustained

BF16 GEMM
619.4 TFLOP/s against 772.9 TFLOP/s
1.25×
FP8 GEMM
1,078.8 TFLOP/s against 1,313.8 TFLOP/s
1.22×
HBM bandwidth
4,012.7 GB/s against 4,464.8 GB/s
1.11×

Throughput as the load rises

Level at one stream, then the gap opens. Llama 3.1 8B, 1k tokens in and 1k out.

MI300XH200

How many long requests fit

Both hold 70 GB of FP8 weights, and what is left is cache. Past its count a GPU queues and preempts.

Concurrent requests of 16k tokens that the KV cache can hold, on Llama 3.3 70B FP8.

MI300X 20.45, from 102.25 GiB of cache
H200 11.12, from 55.60 GiB of cache

From the vLLM server logs on each machine, at identical settings: --max-model-len 16384 and --gpu-memory-utilization 0.90. A production H200 deployment would reduce context length to compensate, at a cost in capability rather than throughput.

First-token wait, training and energy

The MI300X's widest measured lead is the wait for a first token on the 70B at concurrency 256. Training and energy go to the H200.

MI300XH200

First token at p99, 1k in, 1k out
Llama 3.3 70B FP8, concurrency 256. Shorter is better.
MI300X 5.5× sooner
MI300X 26.3 s
H200 145.6 s
First token at p99, 1k in, 8k out
Llama 3.3 70B FP8, concurrency 256. Shorter is better.
MI300X 62.9× sooner
MI300X 35.0 s
H200 2,200.5 s
Training, eager
8B, BF16, tokens per second
H200 1.32× faster
MI300X 5,101.6
H200 6,707
Training, compiled
8B, BF16, tokens per second
H200 1.40× faster
MI300X 5,603.3
H200 7,821.6
Tokens per joule
8B, 1k in and 1k out, concurrency 256
MI300X 10.78
H200 16.37

Every numberThe winners, the data sheets, the KV cache and the full load sweep as tablesShow

Workload	Winner	Margin
Mixture-of-experts decode, concurrency 1	H200	2.72×
Dense 8B, prefill-heavy, concurrency 256	H200	2.06×
Dense 8B, balanced, concurrency 256	H200	1.40×
Dense 8B, single stream	Tie	1.02×
Dense 70B FP8, single stream	H200	1.12×
Dense 70B FP8, balanced, concurrency 128	MI300X	1.14×
Dense 70B FP8, decode-heavy, concurrency 256	MI300X	1.08×
Dense 70B BF16 (141 GB weights)	MI300X	H200 cannot load it
Training, 8B, BF16	H200	1.32× to 1.40×

Specification	MI300X	H200	Paper advantage
BF16 dense	1,307.4 TFLOP/s	989.4 TFLOP/s	MI300X 1.32×
FP8 dense	2,614.9 TFLOP/s	1,978.9 TFLOP/s	MI300X 1.32×
HBM bandwidth	5.3 TB/s	4.8 TB/s	MI300X 1.10×
Capacity	192 GB	141 GB	MI300X 1.36×

Measured best sustained	MI300X	H200	Actual result
BF16 GEMM	619.4 TFLOP/s	772.9 TFLOP/s	H200 1.25×
FP8 GEMM	1,078.8 TFLOP/s	1,313.8 TFLOP/s	H200 1.22×
HBM bandwidth	4,012.7 GB/s	4,464.8 GB/s	H200 1.11×

	Available KV cache	KV cache tokens	Max concurrency at 16k tokens
MI300X	102.25 GiB	335,040	20.45×
H200	55.60 GiB	182,176	11.12×

Concurrency	MI300X, tok/s	H200, tok/s	H200 ratio	MI300X, ms per token	H200, ms per token
1	195	199	1.02×	4.99	5.00
4	603	761	1.26×	6.45	5.21
8	1,137	1,445	1.27×	6.73	5.46
16	2,066	2,674	1.29×	7.42	5.87
32	3,443	4,486	1.30×	9.07	7.00
64	4,956	6,806	1.37×	12.64	9.21
128	7,022	9,937	1.42×	18.00	12.59
256	8,085	11,337	1.40×	30.97	21.55

At concurrency 1 all three 8B request shapes are within 2%, so on this model the gaps are throughput-under-load gaps. Busy-window power at concurrency 256 was 750.3 W on the MI300X and 692.7 W on the H200.

How we ran it

Machines: One MI300X on a DigitalOcean GPU droplet, 191.7 GiB HBM3, 750 W limit. One H200 on Nebius, 140.4 GiB HBM3e, 700 W limit, MIG disabled.
Software: vLLM 0.26.0 and PyTorch 2.11.0 on both sides. The intended difference is ROCm against CUDA.
Numbers: 164 serving data points, 0 dropped for missing telemetry. All figures are medians or percentiles over a fixed grid with 1 Hz telemetry attached. None are best-of-N.

Limits

A virtual function: The MI300X is an SR-IOV virtual function, not bare metal. It presents 304 CUs, 192 GB, SPX/NPS1 and the full 750 W limit, and sustains 100% utilization at 750 W, so it behaves as a whole GPU. Virtualization overhead cannot be excluded, and the H200 is not virtualized the same way. This is the largest caveat.
Different providers: So different host CPUs, RAM, kernels (Ubuntu 26.04 against 24.04) and storage.
One of each: One sample of each GPU. Unit-to-unit variation is not characterised.
Software: Holding vLLM and PyTorch constant removes the framework variable but not the compiler, kernel-library and driver variable. That difference is part of what is measured.
FP8 is not bit-identical: Both GPUs load one shared BF16 checkpoint and derive their own native FP8, because they implement different encodings. On 16 fixed prompts, 3 matched exactly at FP8 against 10 at BF16, and FP8 diverges 4.7× further on mean logprob.
Reduced grids: Concurrency grids are reduced for the large models. Every sweep records the grid it used.
Training: Throughput is not time to convergence. A stack can be fast per step and converge worse.
Not measured: Several GPUs. Nothing here speaks to scaling, collectives or interconnect.

Read the methodology before quoting these numbers. The full report has the microbenchmarks, the mixture-of-experts model, vendor tuning, training and the corrections made during the study.

Sources

The full report Methodology Code and raw results