Read on the NPU. Where to write depends on the reply.
On a Snapdragon 8 Elite Gen 5 the NPU read prompts 10.5 to 15.9 times faster than the CPU, on all nine models we ran. With 64-token replies, reading on the NPU and writing on the CPU was the fastest of the nine pairings, at 41% to 52% of the CPU's time alone; on Qwen3 4B it ties with the NPU alone. At the conversations' own reply lengths, about six times longer, the phone throttled and the NPU alone was faster on most of the transformers.
Each model ran through every combination of where the prompt is read (prefill) and where the reply is written (decode), in one llama.cpp build, on real multi-turn conversations. Measured on 24 and 25 October 2026, on phones in Qualcomm Device Cloud.
64-token replies
- NPU + CPU67.3 s
- NPU only74.9 s
- CPU only148.5 s
The conversations' own reply lengths
- NPU only447.0 s
- NPU + CPU514.5 s
- CPU only583.7 s
Findings
- Prefill: the NPU is fastest on every model, 10.5 to 15.9 times the CPU and 5.1 to 8.2 times the GPU.
- Decode: the CPU is fastest on eight models and level with the NPU on Qwen3 4B (15.27 against 15.35 tok/s). On the transformers its lead over the NPU shrinks with size, from 1.27 to 1.37 times at 0.6B and 1B to 1.08 times on Llama 3.2 3B. The two LFM2.5 hybrids keep a wider lead: 1.48 times at 1.2B, 1.34 times at 2.6B.
- Long context: past 2,048 tokens of context the NPU decoded faster than the CPU on five transformers (Qwen3 4B: 12.6 against 10.6 tok/s). Four or five turns each, late in replay and so also the hottest: a tendency, not a rule.
- Fastest pairing at 64 tokens: NPU prefill with CPU decode on all nine models, at 41% to 52% of CPU-only time. On Qwen3 4B it ties with NPU only: 0.2% ahead in the first pass, 0.3% behind in the second. On Llama 3.2 3B, NPU only is 7.4% slower. At the conversations' own reply lengths this changes; see below.
- GPU: never the fastest at either phase. On the two LFM2.5 hybrids it decodes faster than the NPU (46.7 against 42.7, 21.6 against 20.7 tok/s); on the seven transformers it decodes slowest.
- Handoff: copying the conversation state from one processor to another took a median 0.6 to 11 ms per turn.
- Second pass: five models ran again with every cell in another order. Each kept its slowest pairing, and the four smaller ones their fastest; cells moved a median 1.2% to 2.9%, at most 6.4%. Qwen3 4B's top two swapped, NPU only taking 149.6 s and NPU + CPU 150.1 s: over both passes, a tie.
| Model | NPU prefill vs CPU | NPU prefill vs GPU | Decode tok/s, CPU / GPU / NPU | Fastest | Fastest vs CPU only | Slowest |
|---|---|---|---|---|---|---|
| Qwen3 0.6B Q4_0 | 12.6× | 5.1× | 63.7 / 30.5 / 47.8 | NPU + CPU | 41% | CPU + GPU |
| Gemma 3 1B Q4_0 | 10.5× | 6.0× | 65.6 / 31.1 / 47.8 | NPU + CPU | 48% | CPU + GPU |
| Llama 3.2 1B Q4_0 | 12.5× | 5.7× | 49.2 / 34.3 / 38.8 | NPU + CPU | 47% | CPU + GPU |
| LFM2.5 1.2B QAD Q4_0, hybrid | 10.5× | 5.4× | 63.2 / 46.7 / 42.7 | NPU + CPU | 46% | CPU + NPU |
| Qwen3 1.7B Q4_0 | 15.1× | 7.4× | 33.1 / 18.7 / 29.9 | NPU + CPU | 45% | CPU + GPU |
| Qwen3 1.7B Q8_0 | 13.4× | 7.6× | 26.6 / 16.6 / 23.6 | NPU + CPU | 52% | CPU + GPU |
| LFM2.5 2.6B QAD Q4_0, hybrid | 11.0× | 6.2× | 27.8 / 21.6 / 20.7 | NPU + CPU | 50% | CPU + NPU |
| Llama 3.2 3B Q4_0 | 13.2× | 7.4× | 20.5 / 14.2 / 19.0 | NPU + CPU | 47% | CPU + NPU |
| Qwen3 4B Q4_0 | 15.9× | 8.2× | 15.27 / 6.28 / 15.35 | NPU + CPU | 42% | CPU + GPU |
64-token replies, first pass. Ratios and decode rates are the mean of the three cells that use each processor. A pairing is named prefill processor first, so NPU + CPU reads the prompt on the NPU and writes the reply on the CPU.
Every pairing, model by model
Rows are the processor that reads the prompt, columns the one that writes the reply. Each cell gives prefill and decode in tokens per second, then the total time for the 33 turns, with 64-token replies, in the first pass. The more cobalt a cell carries, the faster it was for that model; the solid cell is the fastest. Five models ran a second pass in another cell order; under each of them, the largest change in a cell's time.
Qwen3 0.6B Q4_0
First session (r4), 1 Hexagon session. Second pass: cells within 5.5%, same fastest
| prefill decode | CPU | GPU | NPU |
|---|---|---|---|
| CPU | 237 / 59.684.3 s | 234 / 29.0122.7 s | 247 / 48.291.1 s |
| GPU | 568 / 62.155.0 s | 532 / 30.390.5 s | 672 / 47.462.4 s |
| NPU | 3,079 / 69.334.6 s, the fastest pairing | 2,965 / 32.169.9 s | 2,999 / 47.647.6 s |
Gemma 3 1B Q4_0
First session (r4), 1 Hexagon session
| prefill decode | CPU | GPU | NPU |
|---|---|---|---|
| CPU | 332 / 62.970.2 s | 336 / 29.8106.8 s | 350 / 48.378.5 s |
| GPU | 561 / 64.854.4 s | 552 / 30.191.5 s | 658 / 47.962.6 s |
| NPU | 3,664 / 69.333.6 s, the fastest pairing | 3,434 / 33.466.2 s | 3,542 / 47.347.5 s |
Llama 3.2 1B Q4_0
First session (r4), 1 Hexagon session. Second pass: cells within 5.9%, same fastest
| prefill decode | CPU | GPU | NPU |
|---|---|---|---|
| CPU | 234 / 46.493.9 s | 241 / 32.7111.5 s | 251 / 39.099.2 s |
| GPU | 525 / 48.864.9 s | 533 / 33.683.5 s | 537 / 38.675.8 s |
| NPU | 3,017 / 52.543.7 s, the fastest pairing | 3,047 / 36.561.1 s | 3,058 / 38.857.4 s |
LFM2.5 1.2B QAD Q4_0, hybrid
First session (r4), 1 Hexagon session. Second pass: cells within 6.4%, same fastest
| prefill decode | CPU | GPU | NPU |
|---|---|---|---|
| CPU | 304 / 59.475.1 s | 327 / 44.883.9 s | 335 / 42.985.0 s |
| GPU | 626 / 63.252.6 s | 585 / 44.467.6 s | 660 / 42.567.7 s |
| NPU | 3,227 / 67.134.9 s, the fastest pairing | 3,401 / 51.044.7 s | 3,477 / 42.852.1 s |
Qwen3 1.7B Q4_0
First session (r4), 1 Hexagon session. Second pass: cells within 3.7%, same fastest
| prefill decode | CPU | GPU | NPU |
|---|---|---|---|
| CPU | 139 / 32.4148.5 s | 138 / 18.3199.5 s | 141 / 29.6153.8 s |
| GPU | 281 / 32.8105.9 s | 274 / 18.6154.2 s | 295 / 30.0110.1 s |
| NPU | 2,091 / 34.167.3 s, the fastest pairing | 2,115 / 19.3114.7 s | 2,086 / 30.074.9 s |
Qwen3 1.7B Q8_0
Second session (r6), 1 Hexagon session
| prefill decode | CPU | GPU | NPU |
|---|---|---|---|
| CPU | 143 / 26.4160.3 s | 143 / 16.1212.1 s | 147 / 25.0163.8 s |
| GPU | 248 / 26.3127.1 s | 240 / 16.5174.8 s | 274 / 24.8127.8 s |
| NPU | 1,937 / 27.083.8 s, the fastest pairing | 2,010 / 17.1128.7 s | 1,870 / 21.1104.7 s |
LFM2.5 2.6B QAD Q4_0, hybrid
Second session (r7), 1 Hexagon session
| prefill decode | CPU | GPU | NPU |
|---|---|---|---|
| CPU | 135 / 27.2163.1 s | 138 / 21.4181.8 s | 137 / 20.2188.2 s |
| GPU | 238 / 27.7124.3 s | 239 / 21.1147.4 s | 253 / 21.0145.7 s |
| NPU | 1,454 / 28.481.3 s, the fastest pairing | 1,574 / 22.2101.7 s | 1,493 / 21.0106.8 s |
Llama 3.2 3B Q4_0
Second session (r6), 2 Hexagon sessions
| prefill decode | CPU | GPU | NPU |
|---|---|---|---|
| CPU | 85 / 20.3237.7 s | 106 / 16.9232.6 s | 86 / 19.0243.8 s |
| GPU | 167 / 20.5171.0 s | 163 / 12.8232.8 s | 165 / 19.0180.2 s |
| NPU | 1,152 / 20.8111.0 s, the fastest pairing | 1,307 / 12.8173.2 s | 1,202 / 19.0119.2 s |
Qwen3 4B Q4_0
Second session (r6), 2 Hexagon sessions. Second pass: cells within 5.1%, NPU only fastest
| prefill decode | CPU | GPU | NPU |
|---|---|---|---|
| CPU | 54 / 15.3353.5 s | 53 / 6.2557.9 s | 55 / 15.4349.8 s |
| GPU | 102 / 15.0253.7 s | 106 / 6.3443.0 s | 105 / 15.3249.0 s |
| NPU | 813 / 15.5149.5 s, the fastest pairing | 951 / 6.4338.9 s | 801 / 15.4149.9 s |
ExecuTorch on the NPU
- Qwen3 0.6B through ExecuTorch 1.5.1's QNN runner (QNN 2.39, a 2,048-token SM8850 export), the app's other runtime.
- NPU only: ExecuTorch cannot move a conversation between its CPU, GPU and NPU programs, so it can fill only the diagonal, and only its NPU cell ran here.
- The 27 of 33 turns whose whole history fits its window, each run twice; 54 of 54 runs succeeded. The llama.cpp column is its NPU-only cell from the first session.
| Qwen3 0.6B, NPU only | ExecuTorch QNN (run 1 / run 2) | llama.cpp Hexagon |
|---|---|---|
| Decode | 30.3 / 30.4 tok/s | 50.7 tok/s |
| Prefill, first turns (same tokens) | 644 / 656 tok/s | 2,400 tok/s |
| Median time to first token | 333 / 330 ms (re-reads the whole history) | 70 ms (reads new tokens only) |
Same timed decode length and, on first turns, the same prompt tokens; different exports and quantisation, so this compares the two configurations as tested, not the runtimes in general. Same-session control: llama.cpp's NPU-only cell, rerun in the second session starting 29 minutes after ExecuTorch ended, gave 50.3 tok/s decode, 2,242 tok/s first-turn prefill and a 70 ms median first token on the same 27 turns, so the gap persists within one session.
How it was run
The conversations
- Three public conversation sets, the first conversation in each length bucket (short, medium, long, by characters) with at least two user turns:
- WildChat-1M (allenai): real users talking to a deployed assistant.
- UltraChat 200k (HuggingFaceH4,
train_sft): generated multi-turn dialogues. - Daring-Anteater (NVIDIA): mostly generated instruction dialogues.
- Replay: teacher forced, so every earlier assistant turn is the dataset's own reply and all nine cells read exactly the same tokens. Up to six turns per conversation, 8,192-token window, greedy.
- Output: 64 tokens per turn in every cell (the first from the prefill side, then 63 timed decode steps), past an end of turn if the model writes one. The reply-length runs decode the length of the dataset's reply instead, every step timed.
- Not used: ShareGPT52K (the Hugging Face datasets server cannot read it) and MT-Bench (user turns only; the replay needs the dataset's replies).
| Conversation | Length | Characters | Turns replayed | New tokens per turn | Context at last turn |
|---|---|---|---|---|---|
| WildChat-1M #23 | short | 1,378 | 3 | 18 to 197 | 303 |
| WildChat-1M #18 | medium | 9,368 | 6 | 63 to 823 | 2,110 |
| WildChat-1M #40 | long | 14,528 | 6 | 263 to 625 | 3,191 |
| UltraChat 200k #3 | short | 2,085 | 2 | 59 to 162 | 221 |
| UltraChat 200k #0 | medium | 3,087 | 4 | 55 to 209 | 501 |
| UltraChat 200k #2 | long | 16,333 | 4 | 139 to 1,104 | 2,417 |
| Daring-Anteater #8 | short | 1,703 | 2 | 59 to 167 | 226 |
| Daring-Anteater #6 | medium | 8,847 | 3 | 253 to 665 | 1,186 |
| Daring-Anteater #0 | long | 11,525 | 3 | 30 to 985 | 1,532 |
33 turns in all. Token counts are from Qwen3's tokenizer; other models tokenize slightly differently.
The phone
Snapdragon 8 Elite Gen 5 (SM8850, board canoe), 23 GB RAM, Android 16, root shell, on AC power in Qualcomm's rack. The 64-token runs used two reservations; the reply-length runs four more, on three units.
CPU: Qualcomm Oryon, 8 cores in two clusters
| Cores | Type | Max clock | Min clock |
|---|---|---|---|
| cpu6, cpu7 | Prime | 4.608 GHz | 768 MHz |
| cpu0 to cpu5 | Performance | 3.629 GHz | 384 MHz |
All eight report implementer 0x51 (Qualcomm), part 0x002. No efficiency cores. The benchmark used 6 threads.
GPU, NPU and runtime
- GPU
- Adreno 840, OpenCL 3.0 with FP16, 160 MHz to 1.2 GHz in 18 steps. Thermally capped on most turns, as far down as 578 MHz.
- NPU
- Hexagon v81 (HTP). Each session: 8 hardware threads, 8 HVX vector units, 1 HMX matrix unit, 8 MB VTCM.
- Runtime
- llama.cpp
b2e5e9bwithggml-cpu,ggml-openclandggml-hexagon, built with Qualcomm's toolchain image.
Limits
- One or two passes. At 64 tokens five models ran twice and four once. At reply length each run is one order on one unit, and three models ran more than once. The 33 turns of a cell are not independent repeats.
- Throttled. Each cell waits until CPU, GPU and NPU are at or under 55 °C (at 64 tokens up to 300 s; 12 of the first pass's 81 cells and 6 of the second's 45 started without passing), but most turns end with the GPU capped or a CPU ceiling lowered, and at reply length 32 of the 40 cells started with the CPU's ceilings lowered. The rankings describe this phone in its rack, not a hand-held phone.
- NPU cells include CPU work. These models share their output layer with the embedding table, and llama.cpp keeps it on the CPU: about 13% of each decode step's weight bytes on Qwen3 4B, up to 45% on Gemma 3 1B.
- Six reservations. The 64-token runs used two reservations of the same phone and the reply-length runs four more, on three units. Unit, order and heat were not varied apart.
- Not measured: an unprivileged app, cold start, prompts past 3,303 tokens, memory, energy and answer quality; at reply length, GPU decode, CPU or GPU prefill into NPU decode, and GPU prefill on any model but Qwen3 0.6B and 1.7B.
Source: the report (docs/research/snapdragon-pd-matrix.md) and its results (tools/snapdragon/results/2026-10-24-sm8850/) in the OpenWeights repository.