Does the exported window matter?

Answers and tool calls

No consistent directional effect found

Two windows agree on 12 to 19 of 90 replies; the same file run twice agrees on 7 to 15. Grades move both ways between windows by about as much as between repeats.

Speed

Paired medians close; Qwen3 32k slower

Same prompt, same turn, Dimensity 9400: LFM2.5 windows within 2.4% of each other. Qwen3 32k was 5.9% slower on prefill and 4.7% on decode than 2k.

Memory

The one thing the window buys, at load

LFM2.5 pays 25 to 33 KB per token of window; Qwen3-1.7B pays 224 KB, so its 32k export needs over 6 GB before the first prompt and 12 GB phones kill it.

Where the window did matter

At 2k, 14 of 60 long replies hit the window

The 2.6B reasons before it answers. At 2k, prompt plus reply reached the window and was cut; at 4k and above only the reply cap cut them. Grades barely moved.

The full sweep

Screen the endpoints before any full sweep

Compute the KV cost, run the smallest and largest window per model and chip, and add middle windows only where those two diverge or fail.

• Two things the matrix found that it was not looking for. Every compiled model this app had ever run was fed no start-of-text token: the ExecuTorch 1.4.0 runtime adds one only when asked and the app never asked. LFM2.5-1.2B shrugs it off; LFM2.5-2.6B scored 0 of 30 on GSM8K on every chip until the engine wrote the token itself, and every 2.6B cell here is from the fixed build. The 1.2B cells remain BOS-less as measured: comparable among themselves, and a description of that deployed configuration. And a phone that has dozed throttles instrumentation two to five times with wild variance, so the local runner now wakes the phone and holds it awake.
• Two 2.6B conditions. Its own template makes it reason before answering, so under the suite's 640-token cap 16 of 30 GSM8K and 26 of 30 IFEval replies never reached an answer: those rows are cap-censored and kept for their BFCL calls, memory and speed. The rerun at a 2048 cap, GSM8K and IFEval only, is the model's real score: 23 to 28 of 30 and 18 to 21 of 30 on every phone and window, with its same-file repeat moving grades by 2 and its BFCL calls unchanged.
• Qwen3 32k on 12 GB phones. Samsung's Heimdall memory guard killed the test process on the Galaxy S25 Ultra and S24+ after 7 to 12 prompts ("Trigger Global kill before GC, Usage 8.67 GB, Threshold 6 GB" in the logcats). A rerun on the Poco with logcat attached died the same way after 7: MIUI's ActivityManager force-stopped it for "used too many pss resource, pss used 8.65 GB, threshold 6 GB". Only the 16 GB Pixel 10 Pro XL finished. Those cells are marked incomplete and carry no grade or timing.
• Not matched conditions. The Dimensity 9400 ran in hand, cooled below 42 C before each probe load. The 8 Elite, Tensor G5 and Exynos 2400 ran racked on Firebase Test Lab, where phones throttle within about 40 seconds, and had no paired probe: their suite timings are per-cell medians over different replies and do not support a speed claim either way.

• What was run. LFM2.5-1.2B: five windows on the Dimensity 9400, 16k and 32k twice; 4k and 32k on the cloud chips, 4k twice on the Exynos. LFM2.5-2.6B: 2k and 32k on the Dimensity, 2k twice; 4k and 32k on the cloud chips; then GSM8K and IFEval again at a 2048 cap on every one of those cells. Qwen3-1.7B: 2k and 32k everywhere, 2k twice on the Dimensity, 32k twice on the Dimensity to catch the kill. Every file loaded at its own exported window. Exports use ExecuTorch 1.4.0's recipes: 8-bit dynamic activations, 4-bit weights in groups of 32, 8-bit embeddings, fp32 KV cache, prefill chunk 2048.
• Review. Codex reviewed the method twice, before and after the data: same-file repeats, a paired speed probe separate from the task eval, a three-way identity metric counted over BFCL only, memory after load, a second architecture, incomplete cells shown as such, and conclusions bounded by the observed run variation. All are in what you see here.
• Where things are. The note: executorch-window-matrix.md. The tables: window-matrix.md. Raw and graded reports: tools/eval/results. The exports, each window a file: LFM2.5-1.2B, LFM2.5-2.6B, Qwen3-1.7B on Hugging Face.

OpenWeightsFive chips, two runtimesWhat the reruns changedSource on GitHub