LLM inference on datacenter, laptop and phone silicon.
Winner and margin, every lane
Bars grow toward the chip that wins, on a log scale.
Server
- NVIDIA H200 wins
- AMD MI300X wins
The answer flips with model size: 70B plus its KV cache no longer fits in 141 GB. 70B BF16 loads only on the MI300X.
Laptop
- Apple M5 wins
- Snapdragon X2 Elite wins
The Snapdragon wins the CPU and the NPU. Training is 19.5x because PyTorch has no Adreno backend, so the X2 trains on its CPU.
Phone
- Snapdragon 8 Elite wins
- Dimensity 9500s wins
The Snapdragon wins every lane, by 1.6x on GPU decode and 18.5x on GPU prefill. The Dimensity's OpenCL is blocked, so it runs Vulkan only.
First token and per token on five phone chips
Five small models on llama.cpp and ExecuTorch, measured inside the OpenWeights app. Medians over 60 to 90 prompts per cell. Interactive version.
Gemma 3 1B
LFM2.5 1.2B
Qwen3 1.7B
Llama 3.2 3B
SmolLM3 3B
- llama.cpp
- ExecuTorch
Both axes are log scales, shared across the five panels. Hover or tap a mark for its number; the rate in parentheses is the median prefill or decode tokens per second.
Conditions are not matched. The Dimensity 9400 ran in hand under a fan; the other four ran racked in cloud test labs, where phones throttle within about 40 seconds. Single pass, 2026-09-03 and 04.
GPU against ASIC
At 32 concurrent streams, one Inferentia2 cost 2.4x as much per output token as one MI300X. Trainium1 fine-tuned the same 8B model at 68.3% MFU. The comparison, with the training, serving and speculative decoding figures.
Server
AMD MI300X against NVIDIA H200 in inference and training, plus Trainium, Inferentia and TPUs.
| Repository | What was measured | Result |
|---|---|---|
| MI300X-vs-H200 One GPU each, serving and training, three shapes, eight concurrency points | One GPU each, serving and training, three shapes, eight concurrency points | MI300X 1.14x on 70B FP8; H200 1.40x on 8B serving and 1.32 to 1.40x on training |
| qwen3.8-27b-mi300x Qwen3.8-27B served from one MI300X with vLLM behind an authenticated endpoint | Qwen3.8-27B served from one MI300X with vLLM behind an authenticated endpoint | OpenAI-compatible endpoint |
| torchneuronx Llama 3.1 8B LoRA on Trainium1, served by vLLM on Inferentia2 | Llama 3.1 8B LoRA on Trainium1, served by vLLM on Inferentia2 | trn1 to inf2 |
| serverless-inference Scale-to-zero inference on RunPod; model and GPU chosen by shootout | Scale-to-zero inference on RunPod; model and GPU chosen by shootout | RunPod, scale to zero |
| compute-visualizer Roofline and five-way bottleneck analysis for H100 training and inference | Roofline and five-way bottleneck analysis for H100 training and inference | H100 roofline, five bottlenecks |
| will-it-asic Will this model fit a TPU, Trainium, Inferentia, Gaudi or GPU? | Will this model fit a TPU, Trainium, Inferentia, Gaudi or GPU? | TPU, Trainium, Inferentia, Gaudi, GPU |
| pytorch-for-asics De-mystifying PyTorch for ASICs, PyTorch Conference Europe 2026 | De-mystifying PyTorch for ASICs, PyTorch Conference Europe 2026 | conference talk |
| xla-agentic-development Skills for coding agents on TPUs: Pallas, XProf, XLA lowering | Skills for coding agents on TPUs: Pallas, XProf, XLA lowering | Claude Code and Codex plugin |
Laptop
Apple M5 against Snapdragon X2 Elite: same llama.cpp release, byte-identical weights, each chip's own GPU backend.
| Repository | What was measured | Result |
|---|---|---|
| snapdragon-vs-m5 37 tests each: CPU, GPU and NPU inference, training, a 10-minute sustained loop, perplexity | 37 tests each: CPU, GPU and NPU inference, training, a 10-minute sustained loop, perplexity | M5 1.93x on GPU decode; X2 1.10x on CPU |
| mlx-models MLP, CNN and ViT trained from scratch on an M5 Air, then Whisper, CLIP, SigLIP | MLP, CNN and ViT trained from scratch on an M5 Air, then Whisper, CLIP, SigLIP | MLX 0.32 on 24 GB unified memory |
| mlx-agentic-development Does an MLX skills kit help a coding agent? Pre-registered, placebo arm, 250 runs | Does an MLX skills kit help a coding agent? Pre-registered, placebo arm, 250 runs | null result, p = 0.69 |
Phone
Snapdragon 8 Elite against Dimensity 9500s, measured on the phones themselves, no root.
| Repository | What was measured | Result |
|---|---|---|
| snapdragon-vs-mediatek NPU, GPU and CPU inference and training, int8 and fp16, on device | NPU, GPU and CPU inference and training, int8 and fp16, on device | 4,307 vs 1,100 to 1,470 GOPS int8 on the NPU |
| poco-phone-ai-training Is the Dimensity 9500s NPU reachable without root? Through NeuroPilot, yes | Is the Dimensity 9500s NPU reachable without root? Through NeuroPilot, yes | 1.1 to 1.5 TOPS int8 |
| openweights Hugging Face open weights on Android, native Kotlin and llama.cpp, no account | Hugging Face open weights on Android, native Kotlin and llama.cpp, no account | on the Play Store |