Skip to content

Read on the NPU. Where to write depends on the reply.

On a Snapdragon 8 Elite Gen 5 the NPU read prompts 10.5 to 15.9 times faster than the CPU, on all nine models we ran. With 64-token replies, reading on the NPU and writing on the CPU was the fastest of the nine pairings, at 41% to 52% of the CPU's time alone; on Qwen3 4B it ties with the NPU alone. At the conversations' own reply lengths, about six times longer, the phone throttled and the NPU alone was faster on most of the transformers.

Each model ran through every combination of where the prompt is read (prefill) and where the reply is written (decode), in one llama.cpp build, on real multi-turn conversations. Measured on 24 and 25 October 2026, on phones in Qualcomm Device Cloud.

64-token replies

  • NPU + CPU67.3 s
  • NPU only74.9 s
  • CPU only148.5 s
Qwen3 1.7B Q4_0: total time for the 33 conversation turns.

The conversations' own reply lengths

  • NPU only447.0 s
  • NPU + CPU514.5 s
  • CPU only583.7 s
The same turns, the mean of two runs, on a scale of their own. A pairing is named prefill processor first. Shorter is faster.

Findings

  • Prefill: the NPU is fastest on every model, 10.5 to 15.9 times the CPU and 5.1 to 8.2 times the GPU.
  • Decode: the CPU is fastest on eight models and level with the NPU on Qwen3 4B (15.27 against 15.35 tok/s). On the transformers its lead over the NPU shrinks with size, from 1.27 to 1.37 times at 0.6B and 1B to 1.08 times on Llama 3.2 3B. The two LFM2.5 hybrids keep a wider lead: 1.48 times at 1.2B, 1.34 times at 2.6B.
  • Long context: past 2,048 tokens of context the NPU decoded faster than the CPU on five transformers (Qwen3 4B: 12.6 against 10.6 tok/s). Four or five turns each, late in replay and so also the hottest: a tendency, not a rule.
  • Fastest pairing at 64 tokens: NPU prefill with CPU decode on all nine models, at 41% to 52% of CPU-only time. On Qwen3 4B it ties with NPU only: 0.2% ahead in the first pass, 0.3% behind in the second. On Llama 3.2 3B, NPU only is 7.4% slower. At the conversations' own reply lengths this changes; see below.
  • GPU: never the fastest at either phase. On the two LFM2.5 hybrids it decodes faster than the NPU (46.7 against 42.7, 21.6 against 20.7 tok/s); on the seven transformers it decodes slowest.
  • Handoff: copying the conversation state from one processor to another took a median 0.6 to 11 ms per turn.
  • Second pass: five models ran again with every cell in another order. Each kept its slowest pairing, and the four smaller ones their fastest; cells moved a median 1.2% to 2.9%, at most 6.4%. Qwen3 4B's top two swapped, NPU only taking 149.6 s and NPU + CPU 150.1 s: over both passes, a tie.
ModelNPU prefill vs CPUNPU prefill vs GPUDecode tok/s, CPU / GPU / NPUFastestFastest vs CPU onlySlowest
Qwen3 0.6B Q4_012.6×5.1×63.7 / 30.5 / 47.8NPU + CPU41%CPU + GPU
Gemma 3 1B Q4_010.5×6.0×65.6 / 31.1 / 47.8NPU + CPU48%CPU + GPU
Llama 3.2 1B Q4_012.5×5.7×49.2 / 34.3 / 38.8NPU + CPU47%CPU + GPU
LFM2.5 1.2B QAD Q4_0, hybrid10.5×5.4×63.2 / 46.7 / 42.7NPU + CPU46%CPU + NPU
Qwen3 1.7B Q4_015.1×7.4×33.1 / 18.7 / 29.9NPU + CPU45%CPU + GPU
Qwen3 1.7B Q8_013.4×7.6×26.6 / 16.6 / 23.6NPU + CPU52%CPU + GPU
LFM2.5 2.6B QAD Q4_0, hybrid11.0×6.2×27.8 / 21.6 / 20.7NPU + CPU50%CPU + NPU
Llama 3.2 3B Q4_013.2×7.4×20.5 / 14.2 / 19.0NPU + CPU47%CPU + NPU
Qwen3 4B Q4_015.9×8.2×15.27 / 6.28 / 15.35NPU + CPU42%CPU + GPU

64-token replies, first pass. Ratios and decode rates are the mean of the three cells that use each processor. A pairing is named prefill processor first, so NPU + CPU reads the prompt on the NPU and writes the reply on the CPU.

Every pairing, model by model

Rows are the processor that reads the prompt, columns the one that writes the reply. Each cell gives prefill and decode in tokens per second, then the total time for the 33 turns, with 64-token replies, in the first pass. The more cobalt a cell carries, the faster it was for that model; the solid cell is the fastest. Five models ran a second pass in another cell order; under each of them, the largest change in a cell's time.

Qwen3 0.6B Q4_0

First session (r4), 1 Hexagon session. Second pass: cells within 5.5%, same fastest

prefill
decode
CPUGPUNPU
CPU237 / 59.684.3 s234 / 29.0122.7 s247 / 48.291.1 s
GPU568 / 62.155.0 s532 / 30.390.5 s672 / 47.462.4 s
NPU3,079 / 69.334.6 s, the fastest pairing2,965 / 32.169.9 s2,999 / 47.647.6 s

Gemma 3 1B Q4_0

First session (r4), 1 Hexagon session

prefill
decode
CPUGPUNPU
CPU332 / 62.970.2 s336 / 29.8106.8 s350 / 48.378.5 s
GPU561 / 64.854.4 s552 / 30.191.5 s658 / 47.962.6 s
NPU3,664 / 69.333.6 s, the fastest pairing3,434 / 33.466.2 s3,542 / 47.347.5 s

Llama 3.2 1B Q4_0

First session (r4), 1 Hexagon session. Second pass: cells within 5.9%, same fastest

prefill
decode
CPUGPUNPU
CPU234 / 46.493.9 s241 / 32.7111.5 s251 / 39.099.2 s
GPU525 / 48.864.9 s533 / 33.683.5 s537 / 38.675.8 s
NPU3,017 / 52.543.7 s, the fastest pairing3,047 / 36.561.1 s3,058 / 38.857.4 s

LFM2.5 1.2B QAD Q4_0, hybrid

First session (r4), 1 Hexagon session. Second pass: cells within 6.4%, same fastest

prefill
decode
CPUGPUNPU
CPU304 / 59.475.1 s327 / 44.883.9 s335 / 42.985.0 s
GPU626 / 63.252.6 s585 / 44.467.6 s660 / 42.567.7 s
NPU3,227 / 67.134.9 s, the fastest pairing3,401 / 51.044.7 s3,477 / 42.852.1 s

Qwen3 1.7B Q4_0

First session (r4), 1 Hexagon session. Second pass: cells within 3.7%, same fastest

prefill
decode
CPUGPUNPU
CPU139 / 32.4148.5 s138 / 18.3199.5 s141 / 29.6153.8 s
GPU281 / 32.8105.9 s274 / 18.6154.2 s295 / 30.0110.1 s
NPU2,091 / 34.167.3 s, the fastest pairing2,115 / 19.3114.7 s2,086 / 30.074.9 s

Qwen3 1.7B Q8_0

Second session (r6), 1 Hexagon session

prefill
decode
CPUGPUNPU
CPU143 / 26.4160.3 s143 / 16.1212.1 s147 / 25.0163.8 s
GPU248 / 26.3127.1 s240 / 16.5174.8 s274 / 24.8127.8 s
NPU1,937 / 27.083.8 s, the fastest pairing2,010 / 17.1128.7 s1,870 / 21.1104.7 s

LFM2.5 2.6B QAD Q4_0, hybrid

Second session (r7), 1 Hexagon session

prefill
decode
CPUGPUNPU
CPU135 / 27.2163.1 s138 / 21.4181.8 s137 / 20.2188.2 s
GPU238 / 27.7124.3 s239 / 21.1147.4 s253 / 21.0145.7 s
NPU1,454 / 28.481.3 s, the fastest pairing1,574 / 22.2101.7 s1,493 / 21.0106.8 s

Llama 3.2 3B Q4_0

Second session (r6), 2 Hexagon sessions

prefill
decode
CPUGPUNPU
CPU85 / 20.3237.7 s106 / 16.9232.6 s86 / 19.0243.8 s
GPU167 / 20.5171.0 s163 / 12.8232.8 s165 / 19.0180.2 s
NPU1,152 / 20.8111.0 s, the fastest pairing1,307 / 12.8173.2 s1,202 / 19.0119.2 s

Qwen3 4B Q4_0

Second session (r6), 2 Hexagon sessions. Second pass: cells within 5.1%, NPU only fastest

prefill
decode
CPUGPUNPU
CPU54 / 15.3353.5 s53 / 6.2557.9 s55 / 15.4349.8 s
GPU102 / 15.0253.7 s106 / 6.3443.0 s105 / 15.3249.0 s
NPU813 / 15.5149.5 s, the fastest pairing951 / 6.4338.9 s801 / 15.4149.9 s

At the conversations' own reply lengths

  • Why: every turn above writes 64 tokens. The datasets' replies to those turns are about six times longer: 401 tokens median by Qwen3's tokenizer, up to 1,141 by Gemma's. At that length writing the reply is 84% to 99% of each pairing's time.
  • Run: each turn decodes its own reply's length, every step timed, in up to four pairings per model chosen from the 64-token ranking, on three units of the phone. The cooling gate before each cell waits, up to 120 s here (300 s for Qwen3 0.6B in r9), without cooling anything, and after the first cell the phone rarely got back under 55 °C in that time: 32 of the 40 cells started with the CPU's clock ceilings lowered, most of them to 1.90 and 2.23 GHz with the GPU capped.
  • Throttled, the decoder decides, and it depends on the model. Where both NPU-prefill cells started throttled, NPU + CPU took 10% to 20% longer than NPU only on Qwen3 1.7B (in two orders), its Q8_0 and Llama 3.2 3B even without the CPU's stalls. The two tied on the two smallest transformers, and on Qwen3 4B, over its 33 turns, NPU only led only through the stalls. On the hybrid LFM2.5 1.2B the throttled CPU slowed but stayed ahead, and NPU + CPU won every turn; on LFM2.5 2.6B the two were level in one run, and in another NPU + CPU lost with the CPU's stalls and won without them. Gemma 3 1B's CPU decoded faster than its NPU even capped (51.5 against 46.5 tok/s on matched capped turns, one schedule).
  • CPU stalls: on the four models with the largest weights, Qwen3 4B, Llama 3.2 3B, Qwen3 1.7B Q8_0 and, in one of its three runs, LFM2.5 2.6B, cells decoding on the CPU had single steps of 0.5 to 15.9 s, 54 to 291 s per cell. Each of those cells reached about 107 °C at some turn's end (the stalled turns themselves ended at 94.6 to 106.6 °C); nothing recorded shows the cause. No NPU-only cell stalled, nor any cell of the five smaller models.
  • When NPU + CPU started after a passed gate (Qwen3 0.6B, Gemma 3 1B), it was fastest, at 75% and 76% of CPU-only time, as at 64 tokens. Qwen3 0.6B's same pairing, started throttled on another unit, took 87%.
  • The NPU keeps its pace: its 64-token decode rate predicted its time at reply length within about 5% on seven models (Qwen3 4B over the 20 turns all its cells completed), and where it ran more than once its totals came within 3% of each other. Cells decoding on the CPU took 3% to 61% longer than predicted, and their totals differed by up to 25% between runs.
ModelNPU onlyNPU + CPUCPU onlyNPU + CPU against NPU only
64-token repliesTheir own lengths
Qwen3 0.6B Q4_0r8 and r9, on 2 units292.7 sr8, r9: 0.0% apart281.5 sr8, r9: 18.3% apart346.6 sr8, r9: 1.3% apart−27.4%r8 −11.9%*r9 +4.2%
Gemma 3 1B Q4_0r11288.3 s221.6 s290.0 s−29.3%−23.1%*
Llama 3.2 1B Q4_0r9330.7 s342.0 s390.8 s−23.9%+3.4%
LFM2.5 1.2B QAD Q4_0, hybridr9321.4 s271.1 s315.3 s−33.1%−15.7%
Qwen3 1.7B Q4_0r9 and r9r, on 1 unit447.0 sr9, r9r: 2.7% apart514.5 sr9, r9r: 1.4% apart583.7 sr9, r9r: 25.1% apart−10.2%r9 +12.8%r9r +17.5%
Qwen3 1.7B Q8_0r11528.1 s634.4 s851.8 s−20.0%+20.1%+10.0% without stalls
LFM2.5 2.6B QAD Q4_0, hybridr9, r10 and r11, on 2 units600.8 sr10, r11: 0.3% apart638.8 sr9, r11: 13.2% apart637.0 sr10, r11: 23.7% apart−23.8%r9 −2.1%, 29 turnsr11 +12.7%−6.9% without stalls
Llama 3.2 3B Q4_0r9733.5 s1,014.1 s1,150.9 s−6.9%+38.2%+19.8% without stalls
Qwen3 4B Q4_0r81,064.7 s1,339.9 scut short, 20 of 33 turns−0.2%+25.8%−1.4% without stalls

r8 to r11 are the report's labels for the runs, each one order on one unit. Seconds are summed over the 33 turns; where a pairing completed them in more than one run, they are the mean of the runs named under it, with how far apart those were. LFM2.5 2.6B's NPU only in r9, cut at 29 turns, is left out of its mean, and r10 has no NPU + CPU: the phone stopped answering before it finished. The last two columns give how much longer NPU + CPU took than NPU only (minus: less): with 64-token replies in the first pass, and at reply length within each run, on the turns both cells completed. * NPU + CPU started after a passed cooling gate and NPU only after a timed-out one. Matching gate results are not matched thermal histories either: a gate that timed out covers a range of temperatures and caps. “Without stalls” deletes the stalled steps' time: it measures their share, not what the cell would have taken without whatever caused them. GPU + CPU ran on Qwen3 0.6B and 1.7B only, at 94% to 100% of CPU-only time.

ExecuTorch on the NPU

  • Qwen3 0.6B through ExecuTorch 1.5.1's QNN runner (QNN 2.39, a 2,048-token SM8850 export), the app's other runtime.
  • NPU only: ExecuTorch cannot move a conversation between its CPU, GPU and NPU programs, so it can fill only the diagonal, and only its NPU cell ran here.
  • The 27 of 33 turns whose whole history fits its window, each run twice; 54 of 54 runs succeeded. The llama.cpp column is its NPU-only cell from the first session.
Qwen3 0.6B, NPU onlyExecuTorch QNN (run 1 / run 2)llama.cpp Hexagon
Decode30.3 / 30.4 tok/s50.7 tok/s
Prefill, first turns (same tokens)644 / 656 tok/s2,400 tok/s
Median time to first token333 / 330 ms (re-reads the whole history)70 ms (reads new tokens only)

Same timed decode length and, on first turns, the same prompt tokens; different exports and quantisation, so this compares the two configurations as tested, not the runtimes in general. Same-session control: llama.cpp's NPU-only cell, rerun in the second session starting 29 minutes after ExecuTorch ended, gave 50.3 tok/s decode, 2,242 tok/s first-turn prefill and a 70 ms median first token on the same 27 turns, so the gap persists within one session.

How it was run

The conversations

  • Three public conversation sets, the first conversation in each length bucket (short, medium, long, by characters) with at least two user turns:
    • WildChat-1M (allenai): real users talking to a deployed assistant.
    • UltraChat 200k (HuggingFaceH4, train_sft): generated multi-turn dialogues.
    • Daring-Anteater (NVIDIA): mostly generated instruction dialogues.
  • Replay: teacher forced, so every earlier assistant turn is the dataset's own reply and all nine cells read exactly the same tokens. Up to six turns per conversation, 8,192-token window, greedy.
  • Output: 64 tokens per turn in every cell (the first from the prefill side, then 63 timed decode steps), past an end of turn if the model writes one. The reply-length runs decode the length of the dataset's reply instead, every step timed.
  • Not used: ShareGPT52K (the Hugging Face datasets server cannot read it) and MT-Bench (user turns only; the replay needs the dataset's replies).
ConversationLengthCharactersTurns replayedNew tokens per turnContext at last turn
WildChat-1M #23short1,378318 to 197303
WildChat-1M #18medium9,368663 to 8232,110
WildChat-1M #40long14,5286263 to 6253,191
UltraChat 200k #3short2,085259 to 162221
UltraChat 200k #0medium3,087455 to 209501
UltraChat 200k #2long16,3334139 to 1,1042,417
Daring-Anteater #8short1,703259 to 167226
Daring-Anteater #6medium8,8473253 to 6651,186
Daring-Anteater #0long11,525330 to 9851,532

33 turns in all. Token counts are from Qwen3's tokenizer; other models tokenize slightly differently.

The phone

Snapdragon 8 Elite Gen 5 (SM8850, board canoe), 23 GB RAM, Android 16, root shell, on AC power in Qualcomm's rack. The 64-token runs used two reservations; the reply-length runs four more, on three units.

CPU: Qualcomm Oryon, 8 cores in two clusters

CoresTypeMax clockMin clock
cpu6, cpu7Prime4.608 GHz768 MHz
cpu0 to cpu5Performance3.629 GHz384 MHz

All eight report implementer 0x51 (Qualcomm), part 0x002. No efficiency cores. The benchmark used 6 threads.

GPU, NPU and runtime

GPU
Adreno 840, OpenCL 3.0 with FP16, 160 MHz to 1.2 GHz in 18 steps. Thermally capped on most turns, as far down as 578 MHz.
NPU
Hexagon v81 (HTP). Each session: 8 hardware threads, 8 HVX vector units, 1 HMX matrix unit, 8 MB VTCM.
Runtime
llama.cpp b2e5e9b with ggml-cpu, ggml-opencl and ggml-hexagon, built with Qualcomm's toolchain image.

Limits

  • One or two passes. At 64 tokens five models ran twice and four once. At reply length each run is one order on one unit, and three models ran more than once. The 33 turns of a cell are not independent repeats.
  • Throttled. Each cell waits until CPU, GPU and NPU are at or under 55 °C (at 64 tokens up to 300 s; 12 of the first pass's 81 cells and 6 of the second's 45 started without passing), but most turns end with the GPU capped or a CPU ceiling lowered, and at reply length 32 of the 40 cells started with the CPU's ceilings lowered. The rankings describe this phone in its rack, not a hand-held phone.
  • NPU cells include CPU work. These models share their output layer with the embedding table, and llama.cpp keeps it on the CPU: about 13% of each decode step's weight bytes on Qwen3 4B, up to 45% on Gemma 3 1B.
  • Six reservations. The 64-token runs used two reservations of the same phone and the reply-length runs four more, on three units. Unit, order and heat were not varied apart.
  • Not measured: an unprivileged app, cold start, prompts past 3,303 tokens, memory, energy and answer quality; at reply length, GPU decode, CPU or GPU prefill into NPU decode, and GPU prefill on any model but Qwen3 0.6B and 1.7B.

Source: the report (docs/research/snapdragon-pd-matrix.md) and its results (tools/snapdragon/results/2026-10-24-sm8850/) in the OpenWeights repository.