63 to 71 ms
per token on Inferentia2 from 1 to 32 streams, while throughput grows 26×
ASICs, July and August 2026
Llama 3.1 8B on AWS Trainium and Inferentia.
A fine-tune on Trainium, served on Inferentia2, measured beside an MI300X and an H200.
What we found
A single GPU was cheaper per token
1.8× to 4.6× cheaper here. At 32 concurrent streams, the highest load all three chips ran, at on-demand list prices, a million output tokens cost $0.51 on Inferentia2, $0.21 on MI300X and $0.28 on H200.
Decode stayed flat under load
Inferentia2 held 63 to 71 ms per token across the sweep while aggregate throughput grew 26×. It stops at 32 streams because 32 GB of device memory holds about 48 resident sequences at 2048 context.
Fine-tuning fits, and some training does not
LoRA or full SFT on a supported architecture with static shapes is a strong fit. Online RL of any kind is blocked.
Most of the walls were toolchain, not silicon
An exporter allowlist, ahead-of-time compilation per tensor shape, and a training class without generate().
Nothing drooped over 30 minutes
At concurrency 8, Inferentia2 kept 100.4% of its first iteration's throughput, 118.8 to 119.3 tok/s.
The results
Serving one 8B model, and what a token costs
$ per 1M output tokens, hourly rate divided by tokens per hour. On-demand list prices.
Inferentia2 stops at 32 streams because 32 GB of device memory holds about 48 resident sequences at 2048 context, and the KV budget is spent. Its decode latency stays flat across the sweep, 63 to 71 ms per token, while aggregate throughput grows 26x. Compared at 32 streams, an Inferentia2 token cost 2.4x an MI300X token and 1.8x an H200 token. At each side's best point the gap was 5.7x and 4.6x. No GPU ran on AWS in either study, so the like-for-like rental (g6e.xlarge, one L40S, $1.86/hr) is the open comparison.
| What differs | Neuron study | GPU study |
|---|---|---|
| vLLM | 0.16 on Neuron, the newest DLAMI that boots on NeuronCore-v2 | 0.26 on the GPUs |
| PyTorch | 2.9.1 | 2.11.0 |
| Cloud | AWS us-west-2, on-demand list price | DigitalOcean (MI300X), Nebius (H200), published GPU-hour rates |
| Device memory | 32 GB | 192 GB (MI300X), 141 GB (H200) |
| Run date | 2026-07 | 2026-08 |
Training on one Trainium1 and one Trainium2
Steady-state tok/s. Hover a bar for MFU.
| Objective, one Trainium1 | Status | Measured |
|---|---|---|
| SFT, LoRA Llama 3.1 8B | works | 2,952 tok/s, 68.3% MFU at seq 2048 |
| ORPO Llama 3.1 8B | works | 1,181 tok/s, 30.2% MFU at length 1024 |
| Pretraining 362M, SmolLM2 shape | runs, unresolved | 4,573 tok/s, 7.0% MFU; the hand-written XLA loop recompiles every step |
| DPO Llama 3.1 8B | unresolved | the reference forward compiles outside the step and the lane dies in a host transfer |
| GRPO, RLVR Qwen3 1.7B | blocked | the training model class has no generate() |
Speculative decoding on one Trainium1
Compile cost, quality and stability
Boot time to first token, inf2
Every graph is compiled ahead of time. Same graphs, new weights: nothing recompiled.
Held-out loss after the fine-tune
| Trainium1 | 2.149 to 1.251 |
| Trainium2 | 2.148 to 1.265 |
Byte-identical held-out rows on both chips. Starting losses agree to three decimals, which is what makes the two runs comparable.
30 minutes of sustained load, inf2
- Retention vs first iteration
- 100.4%
- Throughput, first to last
- 118.8 to 119.3 tok/s
- First-token p99 spread
- 1.6%
Concurrency 8, seven iterations. No thermal or stability droop at this load.
Where it fit, on this evidence
| Workload | Verdict |
|---|---|
| LoRA or full SFT, supported architecture, static shapes | strong fit |
| Preference optimisation without a reference model (ORPO) | works |
| Pretraining a small model from scratch | works to about 400M parameters on one small instance |
| Online RL of any kind (GRPO, PPO, RLVR) | blocked |
| Cost-optimised 8B serving | a single GPU was 1.8× to 4.6× cheaper per token here |
| Serving where capacity or data residency decides | reasonable, inside the concurrency ceiling |
| Architecture outside the exporter list | a wall, not a tuning problem |
How we ran it
- Machines
- One trn1.2xlarge fine-tunes the model, one inf2.xlarge serves it, and one trn2.3xlarge repeats the fine-tune.
- GPU points
- They come from the MI300X and H200 study, which used the same request shape and metric schema.
- Evidence
- Every number has a committed log in torchneuronx. The instances were terminated on 2026-08-26, and the analysis re-runs from the committed results with no AWS account.
Limits
- Not matched conditions
- The Neuron study and the GPU study differ in the ways listed in the table under the first figure.
- Prices
- Cost per token is the hourly rate divided by tokens per hour, at on-demand list prices.
The talk covers the same material.