Five chips, two runtimes.

How fast five small models answer on five phone chips, in llama.cpp and in ExecuTorch.

What we found

18 of 25 pairs reached the first token sooner on ExecuTorch
Time to first token is what a user waits before the reply starts. Across both runtimes it ran from 0.30 s to 15.90 s.
19 of 25 streamed faster on ExecuTorch
Time per output token is how fast the reply then streams. Tensor G5 and Exynos 2400 decode at about half speed on llama.cpp against ExecuTorch, cause unknown.
SmolLM3 is the exception on prefill
On ExecuTorch it prefills at a third of llama.cpp's rate on every chip, which is the compiled model rather than the phones.

The results

Every model on every chip

Latency map

Each line joins one chip's two runtimes. Lower and left is faster. Both axes are logarithmic.

llama.cpp, GGUFExecuTorch

All models, all chips

The same numbers as a grid, both columns on logarithmic scales. The model picked above is banded.

llama.cpp, GGUFExecuTorch

Every number

Medians per cell, llama.cpp beside ExecuTorch. Bold blue marks the faster runtime on that chip. Prefill is the median rate behind the time to first token.

Chip	Time to first token, s		Prefill, tok/s		Time per token, ms
Chip	llama.cpp	ExecuTorch	llama.cpp	ExecuTorch	llama.cpp	ExecuTorch
Gemma 3 1B
Dimensitys	1.27	0.43, faster	61	164, faster	42, faster	48
Snapdragon 8 Gen 3	1.02	0.37, faster	69	192, faster	51	27, faster
Snapdragon 8 Elite	0.80	0.40, faster	79	167, faster	28, faster	44
Tensor G5	1.35	0.48, faster	56	156, faster	113	50, faster
Exynos 2400	1.68	0.45, faster	45	151, faster	75	56, faster
LFM2.5 1.2B
Dimensitys	0.81	0.30, faster	138	321, faster	31	25, faster
Snapdragon 8 Gen 3	0.73	0.44, faster	130	217, faster	43	27, faster
Snapdragon 8 Elite	0.59	0.30, faster	153	288, faster	27	18, faster
Tensor G5	0.83	0.32, faster	112	248, faster	106	46, faster
Exynos 2400	0.97	0.42, faster	92	208, faster	59	38, faster
Qwen3 1.7B
Dimensitys	1.05	0.76, faster	113	122, faster	57, faster	64
Snapdragon 8 Gen 3	0.49, faster	0.77	195, faster	120	74	41, faster
Snapdragon 8 Elite	0.37, faster	0.75	266, faster	130	40, faster	41
Tensor G5	1.17	0.89, faster	93	102, faster	184	94, faster
Exynos 2400	0.89	0.81, faster	100	112, faster	97	67, faster
Llama 3.2 3B
Dimensitys	2.39	1.36, faster	39	87, faster	82	82
Snapdragon 8 Gen 3	2.11	1.82, faster	43	66, faster	107	91, faster
Snapdragon 8 Elite	1.70	1.47, faster	48	80, faster	83	63, faster
Tensor G5	2.43	1.55, faster	37	71, faster	180	125, faster
Exynos 2400	3.46	1.73, faster	25	64, faster	151	119, faster
SmolLM3 3B
Dimensitys	1.18, faster	12.69	42, faster	13	79, faster	91
Snapdragon 8 Gen 3	1.43, faster	10.33	39, faster	14	101	85, faster
Snapdragon 8 Elite	1.03, faster	7.18	47, faster	21	79	67, faster
Tensor G5	1.86, faster	15.90	35, faster	11	239	108, faster
Exynos 2400	1.79, faster	12.26	29, faster	11	132	121, faster

How we ran it

Prompts: The same 90 prompts: 30 GSM8K, 30 IFEval and 30 BFCL, except that Gemma 3 skips BFCL, having no tool syntax. So 60 to 90 per cell, 19 to 772 tokens long and 92 at the median, run greedy with thinking off.
Files: llama.cpp ran Q4_K_M GGUFs, Qwen3 at Q8_0. ExecuTorch ran the published XNNPACK INT8/INT4 exports, LFM2.5 as 8da4w and Llama 3.2 as SpinQuant.
Runs: One pass per cell with no repeats, on 3 and 4 September 2026.
Phone time: 29 hours in all, plus loading and Test Lab queueing. Generation time per cell ran from 6 minutes (LFM2.5-1.2B on ExecuTorch, 8 Elite) to about 100 minutes (SmolLM3 on llama.cpp, Tensor G5).

Limits

What it is for: Picking a model, and one of the two runtimes the app ships, for a given phone.
Timing only: This page has no quality numbers. The pass rates on the same prompts are in the repository's benchmark matrix.
Not matched conditions: The Dimensitys ran in hand under a desk fan. The Snapdragon 8 Gen 3 ran on Qualcomm Device Cloud and the 8 Elite, Tensor G5 and Exynos 2400 on Firebase Test Lab, racked, where phones throttle within about 40 seconds.

Sources

Benchmark matrix, with pass rates OpenWeights