Llama 3.1 8B on AWS Trainium and Inferentia.

One trn1.2xlarge fine-tunes the model, one inf2.xlarge serves it, one trn2.3xlarge repeats the fine-tune, and every number has a committed log in torchneuronx. The GPU points come from MI300X-vs-H200, which used the same request shape and metric schema. The conditions that differ are listed under the first figure. The talk covers the same material.

Serving one 8B model, and what a token costs

measured on all three101001,00010,000tok/sH200MI300XInferentia211 concurrent streamH200 199 tok/sMI300X 195 tok/sInferentia2 15.7 tok/s44 concurrent streamsH200 761 tok/sMI300X 603 tok/sInferentia2 61.4 tok/s88 concurrent streamsH200 1,445 tok/sMI300X 1,137 tok/sInferentia2 119.3 tok/s1616 concurrent streamsH200 2,674 tok/sMI300X 2,066 tok/sInferentia2 226.6 tok/s3232 concurrent streamsH200 4,486 tok/sMI300X 3,443 tok/sInferentia2 415.5 tok/s6464 concurrent streamsH200 6,806 tok/sMI300X 4,956 tok/s128128 concurrent streamsH200 9,937 tok/sMI300X 7,022 tok/s256256 concurrent streamsH200 11,337 tok/sMI300X 8,085 tok/sConcurrent streams. Llama 3.1 8B Instruct, BF16, vLLM, 1k in / 1k out, output tok/s over all streams.
at 32 streams best observed
inf2.xlarge, 1x Inferentia2
$0.507415.5 tok/s at 32 streams, $0.7582/hr
$0.507415.5 tok/s at 32 streams, $0.7582/hr
MI300X, 1 GPU
$0.2093,443 tok/s at 32 streams, $2.59/hr
$0.0898,085 tok/s at 256 streams, $2.59/hr
H200, 1 GPU
$0.2794,486 tok/s at 32 streams, $4.5/hr
$0.11011,337 tok/s at 256 streams, $4.5/hr

$ per 1M output tokens, hourly rate divided by tokens per hour. On-demand list prices.

Inferentia2 stops at 32 streams because 32 GB of device memory holds about 48 resident sequences at 2048 context, and the KV budget is spent. Its decode latency stays flat across the sweep, 63 to 71 ms per token, while aggregate throughput grows 26x. Compared at 32 streams, an Inferentia2 token cost 2.4x an MI300X token and 1.8x an H200 token. At each side's best point the gap was 5.7x and 4.6x. No GPU ran on AWS in either study, so the like-for-like rental (g6e.xlarge, one L40S, $1.86/hr) is the open comparison.

What differsNeuron studyGPU study
vLLM0.16 on Neuron, the newest DLAMI that boots on NeuronCore-v20.26 on the GPUs
PyTorch2.9.12.11.0
CloudAWS us-west-2, on-demand list priceDigitalOcean (MI300X), Nebius (H200), published GPU-hour rates
Device memory32 GB192 GB (MI300X), 141 GB (H200)
Run date2026-072026-08

Training on one Trainium1 and one Trainium2

Trainium1, $1.34/hr Trainium2, $2.24/hr
2,952
Trainium1, seq 2,048: 2,952 tok/s, 68.3% MFU
3,618
Trainium2, seq 2,048: 3,618 tok/s, 26.5% MFU
3,575
Trainium1, seq 4,096: 3,575 tok/s, 82.7% MFU
7,340
Trainium2, seq 4,096: 7,340 tok/s, 50.3% MFU
8,337
Trainium2, seq 8,192: 8,337 tok/s, 61% MFU
seq 2,048seq 4,096seq 8,192

Steady-state tok/s. Hover a bar for MFU.

Objective, one Trainium1StatusMeasured
SFT, LoRA
Llama 3.1 8B
works2,952 tok/s, 68.3% MFU at seq 2048
ORPO
Llama 3.1 8B
works1,181 tok/s, 30.2% MFU at length 1024
Pretraining
362M, SmolLM2 shape
runs, unresolved4,573 tok/s, 7.0% MFU; the hand-written XLA loop recompiles every step
DPO
Llama 3.1 8B
unresolvedthe reference forward compiles outside the step and the lane dies in a host transfer
GRPO, RLVR
Qwen3 1.7B
blockedthe training model class has no generate()
Llama 3.1 8B Instruct, LoRA r16, bf16, micro-batch 1, Trainium1 at TP=2 and Trainium2 at TP=4, the working default of each chip. At sequence 2048 the Trainium2 is 1.2x faster and runs at 26.5% MFU: the step is too small to fill a chip with 3.2x the peak. At 4096 it is 2x faster, and 8192 fits only on the Trainium2. Two physical Trainium2 chips ran the study and differed by 2.4%, the same as the seed-to-seed noise floor, so the final loss was bit-identical across chips (1.1489). The Trainium1 fine-tune cost $0.126 per 1M training tokens at 2048 and $0.104 at 4096. The GPU study trained the full 8B in BF16, a different quantity, so no GPU training number is placed beside these.

Speculative decoding on one Trainium1

030609060%80%100%tok/saccepted0no draft31.61 tok/s, 1.00xtarget only22 draft tokens per call44.51 tok/s, 1.41x96.6% of drafts accepted33 draft tokens per call56.41 tok/s, 1.78x92.8% of drafts accepted44 draft tokens per call64.3 tok/s, 2.03x88.8% of drafts accepted55 draft tokens per call70.37 tok/s, 2.23x85.9% of drafts accepted66 draft tokens per call73.86 tok/s, 2.34x82.4% of drafts accepted77 draft tokens per call77.34 tok/s, 2.45x80.4% of drafts accepted1010 draft tokens per call78.47 tok/s, 2.48x70.9% of drafts acceptedDraft tokens per call. Llama 3.1 8B target, Llama 3.2 1B draft, greedy, 39 prompts, 43k tokens.
Single-stream decode on one Trainium1. The 1B draft proposes k tokens and the 8B target verifies them in one pass, so throughput rises until rejected drafts cost more than they save. Agreement is the share of drafted tokens the target kept, a speed statistic, not an accuracy score.

Compile cost, quality and stability

Boot time to first token, inf2

Llama 3.1 8B, first ever boot
2,372 s39.5 min, cold cache
Same config, weights from the trn1 fine-tune
548 s9.1 min, warm cache, 0 new NEFFs

Every graph is compiled ahead of time. Same graphs, new weights: nothing recompiled.

Held-out loss after the fine-tune

Trainium12.149 to 1.251
Trainium22.148 to 1.265

Byte-identical held-out rows on both chips. Starting losses agree to three decimals, which is what makes the two runs comparable.

30 minutes of sustained load, inf2

Retention vs first iteration
100.4%
Throughput, first to last
118.8 to 119.3 tok/s
First-token p99 spread
1.6%

Concurrency 8, seven iterations. No thermal or stability droop at this load.

Where it fit, on this evidence

LoRA or full SFT, supported architecture, static shapesstrong fit
Preference optimisation without a reference model (ORPO)works
Pretraining a small model from scratchworks to about 400M parameters on one small instance
Online RL of any kind (GRPO, PPO, RLVR)blocked
Cost-optimised 8B servinga single GPU was 1.8x to 4.6x cheaper per token here
Serving where capacity or data residency decidesreasonable, inside the concurrency ceiling
Architecture outside the exporter lista wall, not a tuning problem

Most of the study's walls were toolchain, not silicon: an exporter allowlist, ahead-of-time compilation per tensor shape, and a training class without generate(). The instances were terminated on 2026-08-26; the analysis re-runs from the committed results with no AWS account.