What the reruns changed.
A second pass over the window matrix. 7 September 2026.
After the window matrix was published, every cell was checked for a failed or noisy run, and three things were rerun the same day.
- Purpose
- Settle the three cells the first pass left in doubt: the 2.6B's grades, which its 640-token cap had cut short; the Poco's Qwen3 32k run, which had died without a recorded cause; and whether a second run of the same file on another phone varies as much as it did on the Poco.
- How it was graded
- Same prompts and graders as the matrix: GSM8K by the last number in the reply, IFEval by the benchmark's instruction checkers, BFCL by one call to the right function with the right arguments. Greedy, thinking off. Every run also recorded prefill and decode time per prompt and resident memory, shown in the timing panel and table. The 2.6B rerun raised only the reply cap, from 640 to 2048, on its 60 GSM8K and IFEval prompts; the repeats used all 90.
- How long it ran
Run Where Wall clock Qwen3 32k with a log Poco 2 min, killed 2.6B 2k repeat, 90 prompts Poco 48 min 2.6B 2k at 2048, 60 prompts Poco 80 min 2.6B 32k at 2048, 60 prompts Poco 66 min 2.6B 4k and 32k at 2048, 1.2B 4k repeat Test Lab, 3 phones 22 launches, 12:40 to 18:51 The Test Lab span includes a pause from 13:19 to 15:05 for the daily quota. Generation time alone: 2.6B at 2048, 130 to 213 minutes per phone.
Findings
The 2.6B
The 640-token cap suppressed its grades
GSM8K 10 to 19 of 30 and IFEval 3 to 6 at the 640 cap; 23 to 28 and 18 to 21 at 2048, on every phone and window.
The window
At 2k, 14 of 60 replies reached the window limit
Prompt plus reply stopped at the window size. At 4k and 32k no reply reached the window; the ones still cut reached the 2048 reply cap.
Same file, twice
One cloud repeat, the same low agreement
The Exynos 2400 repeat of the 1.2B agreed with its first run on 7 of 90 raw replies, at least as variable as the Poco controls. The 2.6B repeat moved grades by 2 per set.
Qwen3 32k
Both memory guards report a 6 GB threshold
MIUI stopped the Poco run at 8.65 GB against a 6 GB threshold, as Samsung's Heimdall stopped the two Galaxys. The Pixel, with no such guard, finished at 8.5 GB.
Grades, cuts, repeats, memory and timing
The 2.6B under two reply caps
Prompts passed out of 30, per phone and window. The hollow marker is the 640-token cap the suite uses, the solid one the 2048-token rerun. BFCL was not rerun; its 640-cap grades and calls stay in the original rows.
The chart scrolls sideways on a narrow screen.
Dimensity 9400Snapdragon 8 EliteTensor G5Exynos 2400640 cap2048 cap
Where the long replies were cut
The 60 GSM8K and IFEval replies of each 2048-cap cell: finished, cut by the window (prompt plus reply reached the window size), or cut by the reply cap.
The chart scrolls sideways on a narrow screen.
FinishedCut by the windowCut by the 2048 cap
Reply length at 2k and at 32k
Dimensity 9400, 2048 cap, 60 replies per window, both panels on one scale. Each bar is a 128-token bin of prompt plus reply. At 2k, 14 replies end at the window limit of 2048 tokens; at 32k, 10 of 60 reached the reply cap and the rest finished earlier.
The chart scrolls sideways on a narrow screen.
2048 tokens, the 2k window
Same file, run twice
Five controls: the same export, same prompts, same phone, a second time. How many replies came back byte-identical (raw stream and shown text over 90 prompts, parsed tool calls over the 30 BFCL prompts), and how far each grade moved.
The chart scrolls sideways on a narrow screen.
Raw streamShown textTool calls
Qwen3-1.7B at 32k against the 6 GB rule
Resident memory (RSS) of the test process right after the 32k export loaded, on each phone, with the phone's RAM. The dashed line is the 6 GB threshold the killed runs' logs reported; that threshold is measured in PSS at kill time, a different measure from the bars, so the bars show which phones were near it, not the moment they crossed it. Three of four processes were stopped within the first dozen prompts.
The chart scrolls sideways on a narrow screen.
6 GB, the PSS threshold Samsung Heimdall and MIUI reported when they stopped the processThe phone's RAM
Time to first token and time per output token
Every rerun recorded prefill and decode time per prompt. TTFT here is the runtime's prefill of the whole prompt, the engine's part of the wait before the first token; TPOT is decode time over generated tokens. Each row pairs a rerun with the earlier run of the same cell: the hollow marker is the earlier run, the solid one the rerun. Medians over the prompts each run completed, so they are per-cell figures, not a paired speed test.
What to take from it: the capped rerun's lower TTFT is a prompt-mix effect, not a speed change. Its 60 GSM8K and IFEval prompts are shorter than the 30 BFCL prompts with their tool definitions, which the 640-cap rows include. TPOT, which does not depend on prompt length, agrees within 5% on the Dimensity 9400 and Exynos 2400, differs by 8 to 11% on the 8 Elite and on the Tensor G5 at 4k, and by 26% on the Tensor G5 at 32k (81 then 102 ms), the size of the launch-to-launch swing seen on that phone before. Of the five repeats, three agree within 4% on both figures; the 1.2B repeats at 32k on the Dimensity and at 4k on the Exynos were 7 to 12% slower the second time.
The chart scrolls sideways on a narrow screen.
Dimensity 9400Snapdragon 8 EliteTensor G5Exynos 2400Earlier runRerun
Notes
- Why the 2.6B needed a rerun. Its chat template opens a thinking block before every answer, and the suite runs thinking off with a 640-token cap. Most of its GSM8K and IFEval replies were still reasoning when generation reached the cap, so those grades confound answer quality with the 640-token limit. The rerun keeps the same prompts, graders and thinking-off rule and only raises the cap to 2048. The 640-cap rows stay in the matrix as their own condition.
- What the cut replies mean. A reply cut by the window exceeded the export's prompt-plus-reply length in this test; a reply cut by the cap is one the benchmark chose not to wait for. In every cell 10 to 15 of 60 replies were cut. At 2k on the Poco all 14 reached the window limit and none the cap; in the seven other cells, at 4k and 32k on every phone, 10 to 15 reached the 2048 cap and none the window. Some prompts make this model reason past 2000 tokens.
- The kill rule. Samsung's Heimdall wrote "Trigger Global kill before GC, Usage 8.67 GB, Threshold 6 GB" on the S25 Ultra and S24+. MIUI on the Poco wrote "used too many pss resource, pss used 8653468kb, pss threshold 6291456kb". Different code, same number: on those two phones a process whose PSS passes 6 GB is stopped. The 32k Qwen3 export needs about 7 GB of KV cache before the first prompt; it measured 6.1 to 6.4 GB resident after load on the three runs that were stopped and 8.5 GB on the Pixel run that finished.
- What was not rerun. The cloud phones' suite timings, which differed by up to threefold between two launches of the same Tensor G5 cell and may reflect different thermal states. A repeat would be as launch-dependent; a paired probe on those phones is required to compare their speed.
- Cost. The cloud reruns took 22 Test Lab launches, one set per launch and continued where the 38-minute budget cut them; 20 ran and 2 were refused when the Blaze plan's quota of 50 physical-device tests per day ran out at 13:19. They resumed at 15:05 after the reset and finished at 18:51. The Poco ran its four steps in 3 h 20 min.
The rerun tables
The rerun cells and controls. The full matrix, with the 640-cap rows and the fixed-prompt speed probe, is on the window page.
LFM2.5-2.6B under two reply caps
8 cells, GSM8K and IFEval
Passed out of 30 at the suite's 640-token cap and at the 2048-token rerun, and how the 60 rerun replies ended.
| Phone | Window | GSM8K at 640 | GSM8K at 2048 | IFEval at 640 | IFEval at 2048 | Finished | Cut by window | Cut by cap |
|---|---|---|---|---|---|---|---|---|
| Dimensity 9400 | 2k | 15/30 | 24/30 | 4/30 | 21/30 | 46 | 14 | 0 |
| Dimensity 9400 | 32k | 13/30 | 26/30 | 5/30 | 21/30 | 50 | 0 | 10 |
| Snapdragon 8 Elite | 4k | 13/30 | 25/30 | 5/30 | 19/30 | 49 | 0 | 11 |
| Snapdragon 8 Elite | 32k | 13/30 | 24/30 | 5/30 | 21/30 | 45 | 0 | 15 |
| Tensor G5 | 4k | 17/30 | 23/30 | 6/30 | 21/30 | 47 | 0 | 13 |
| Tensor G5 | 32k | 19/30 | 28/30 | 4/30 | 18/30 | 48 | 0 | 12 |
| Exynos 2400 | 4k | 10/30 | 25/30 | 5/30 | 21/30 | 47 | 0 | 13 |
| Exynos 2400 | 32k | 18/30 | 27/30 | 3/30 | 19/30 | 50 | 0 | 10 |
Same file, run twice
5 controls
Replies byte-identical between the two runs, and each set's grade in the first and the second run.
| Model | Phone | Window | Raw identical | Text identical | Calls identical | GSM8K | IFEval | BFCL |
|---|---|---|---|---|---|---|---|---|
| LFM2.5-1.2B | Dimensity 9400 | 16k | 13/90 | 20/90 | 23/30 | 13 then 14 | 20 then 16 | 27 then 28 |
| LFM2.5-1.2B | Dimensity 9400 | 32k | 15/90 | 18/90 | 25/30 | 12 then 12 | 18 then 15 | 27 then 27 |
| LFM2.5-2.6B | Dimensity 9400 | 2k | 0/90 | 29/90 | 24/30 | 15 then 13 | 4 then 6 | 24 then 24 |
| Qwen3-1.7B | Dimensity 9400 | 2k | 31/90 | 31/90 | 30/30 | 20 then 17 | 15 then 18 | 24 then 24 |
| LFM2.5-1.2B | Exynos 2400 | 4k | 7/90 | 11/90 | 23/30 | 13 then 15 | 17 then 21 | 26 then 27 |
Timing of every rerun cell
TTFT, TPOT, throughput and memory, against the earlier run of the same cell
Medians over the prompts each run completed. Prefill ms is the runtime's prefill of the whole prompt; ms per token is decode time over generated tokens. RSS is resident memory after load and at the end of the run.
| Cell | Run | Prefill ms | ms per token | Prefill tok/s | Decode tok/s | RSS after load MB | RSS at end MB |
|---|---|---|---|---|---|---|---|
| LFM2.5-2.6B 2k, Dimensity 9400 | 640 cap, 90 prompts | 882 | 60.3 | 108 | 16.6 | 1979 | 1958 |
| 2048 cap, 60 prompts | 693 | 62.8 | 105 | 15.9 | 1959 | 1783 | |
| LFM2.5-2.6B 4k, Snapdragon 8 Elite | 640 cap, 90 prompts | 1197 | 55.4 | 72 | 18 | 2058 | 2194 |
| 2048 cap, 60 prompts | 866 | 51.1 | 82 | 19.6 | 2055 | 2139 | |
| LFM2.5-2.6B 4k, Tensor G5 | 640 cap, 90 prompts | 1066 | 92.2 | 88 | 10.8 | 2054 | 2221 |
| 2048 cap, 60 prompts | 776 | 81.9 | 106 | 12.2 | 2055 | 2188 | |
| LFM2.5-2.6B 4k, Exynos 2400 | 640 cap, 90 prompts | 1142 | 83.5 | 81 | 12 | 2136 | 2189 |
| 2048 cap, 60 prompts | 893 | 87.1 | 84 | 11.5 | 2136 | 2187 | |
| LFM2.5-2.6B 32k, Dimensity 9400 | 640 cap, 90 prompts | 882 | 59.7 | 109 | 16.8 | 2937 | 2185 |
| 2048 cap, 60 prompts | 621 | 59.5 | 113 | 16.8 | 2922 | 2262 | |
| LFM2.5-2.6B 32k, Snapdragon 8 Elite | 640 cap, 90 prompts | 1096 | 49.7 | 81 | 20.1 | 2950 | 3082 |
| 2048 cap, 60 prompts | 982 | 54.5 | 80 | 18.4 | 2950 | 2626 | |
| LFM2.5-2.6B 32k, Tensor G5 | 640 cap, 90 prompts | 968 | 81.1 | 101 | 12.3 | 2962 | 3120 |
| 2048 cap, 60 prompts | 842 | 102.4 | 80 | 9.8 | 2962 | 3123 | |
| LFM2.5-2.6B 32k, Exynos 2400 | 640 cap, 90 prompts | 1085 | 84.5 | 80 | 11.8 | 3032 | 3084 |
| 2048 cap, 60 prompts | 924 | 87.8 | 82 | 11.4 | 3028 | 3080 | |
| LFM2.5-1.2B 4k, Exynos 2400 | first run | 460 | 34 | 195 | 29.4 | 1126 | 1213 |
| second run | 516 | 37.5 | 180 | 26.7 | 1126 | 1217 | |
| LFM2.5-1.2B 16k, Dimensity 9400 | first run | 354 | 25.9 | 262 | 38.6 | - | 993 |
| second run | 368 | 26.2 | 259 | 38.2 | 1416 | 1528 | |
| LFM2.5-1.2B 32k, Dimensity 9400 | first run | 366 | 26.3 | 257 | 38 | - | 1903 |
| second run | 392 | 28.8 | 237 | 34.8 | 1798 | 1901 | |
| LFM2.5-2.6B 2k, Dimensity 9400 | first run | 882 | 60.3 | 108 | 16.6 | 1979 | 1958 |
| second run | 872 | 59.6 | 110 | 16.8 | 1984 | 1871 | |
| Qwen3-1.7B 2k, Dimensity 9400 | first run | 532 | 45.1 | 174 | 22.2 | 1758 | 1833 |
| second run | 532 | 45.7 | 174 | 21.9 | 1766 | 1900 |
Qwen3-1.7B 32k on four phones
The kill evidence
| Phone | Device | RAM | Resident after load | Prompts completed | What the log says |
|---|---|---|---|---|---|
| Dimensity 9400 | Poco X8 Pro | 12 GB | 6188 MB | 6 of 90 | MIUI ActivityManager, 6 GB PSS threshold (rerun: 7 prompts, 8.65 GB) |
| Snapdragon 8 Elite | Galaxy S25 Ultra | 12 GB | 6127 MB | 12 of 90 | Samsung Heimdall, 6 GB threshold (8.67 GB used) |
| Exynos 2400 | Galaxy S24+ | 12 GB | 6396 MB | 7 of 90 | Samsung Heimdall, 6 GB threshold |
| Tensor G5 | Pixel 10 Pro XL | 16 GB | 8499 MB | 90 of 90 | finished all 90 |
Sources and terms
- Where the numbers come from. Grades, controls and memory figures are the rendered matrix's. The cut reasons and the reply-length histogram come from the graded JSON reports, which record prompt and reply token counts per prompt; a reply is counted as cut by the window when prompt plus reply reached the window size, and as cut by the cap when the reply alone reached 2048. Phone RAM is the manufacturer's figure for each device. Launch times and counts are from the run logs.
- Terms. A cell is one export on one phone over the prompt set. An export is the model compiled for ExecuTorch at a fixed context window. Greedy means the most likely token is always taken, so a run should repeat exactly. Thinking off means the model is not given room to reason before answering, though the 2.6B's template does so anyway. GSM8K is grade-school maths, IFEval instruction following, BFCL tool calling. Raw stream is every token generated, shown text is what a user would see after tool-call markup is removed, parsed tool calls are the function calls the app would act on. TTFT is time to first token, measured here as the runtime's prefill of the prompt; TPOT is time per output token, decode time over generated tokens. RSS is the process's resident memory; PSS is its proportional share, the figure the memory guards read. The KV cache is the memory a model keeps per token of window. Heimdall is Samsung's memory guard. Test Lab is Firebase's rented-phone service.
- Where things are. Raw and graded reports under tools/eval/results: the capped rerun as
-cap2048files, continued sets as@n, second runs under arepeat-prefix, the Poco kill evidence underqwen3-32k-poco-rerun/. The note: executorch-window-matrix.md. The exports on Hugging Face: LFM2.5-2.6B, LFM2.5-1.2B, Qwen3-1.7B.
OpenWeightsDoes the exported window matter?Five chips, two runtimesSource on GitHub