About 4×

the median IFEval gain from raising the reply cap to 2048 tokens: 18 to 21 of 30, up from 3 to 6

OpenWeights, 7 September 2026

A 640-token reply cap was cutting off the LFM2.5 2.6B.

The model reasons before it answers, so a 640-token cap cut it off. Raising only the cap lifted its scores on every phone. The same day's reruns also tested repeats of one file and a run that had died.

What we found

The 640-token cap held the 2.6B back
At 2048 it scored 23 to 28 of 30 on GSM8K and 18 to 21 on IFEval, against 10 to 19 and 3 to 6.
At 2k, 14 of 60 replies hit the window
At 4k and 32k none did; the long ones stopped at the 2048 reply cap instead.
Repeats still disagree
The Exynos 2400 repeat of the 1.2B matched its first run on 7 of 90 raw replies.
Two memory guards, one 6 GB limit
The stopped runs' logs, from MIUI and from Samsung's Heimdall, both report a 6 GB threshold. The Pixel run finished all 90 prompts.

The results

Grades, cuts, repeats, memory and timing

The 2.6B under two reply caps

Prompts passed out of 30, per phone and window. The hollow marker is the 640-token cap the suite uses, the solid one the 2048-token rerun. BFCL was not rerun; its 640-cap grades and calls stay in the original rows.

The chart scrolls sideways on a narrow screen.

DimensitysSnapdragon 8 EliteTensor G5Exynos 2400640 cap2048 cap

Where the long replies were cut

The 60 GSM8K and IFEval replies of each 2048-cap cell: finished, cut by the window (prompt plus reply reached the window's limit), or cut by the reply cap.

The chart scrolls sideways on a narrow screen.

FinishedCut by the windowCut by the 2048 cap

Reply length at 2k and at 32k

Dimensitys, 2048 cap, 60 replies per window, both panels on one scale. Each bar is a 128-token bin of prompt plus reply. At 2k, 14 replies end at the window limit, 2,047 tokens in all; at 32k, 10 of 60 reached the reply cap and the rest finished earlier.

The chart scrolls sideways on a narrow screen.

2048 tokens, the 2k window

Same file, run twice

Five controls: the same export, same prompts, same phone, a second time. How many replies came back byte-identical (raw stream and shown text over 90 prompts, parsed tool calls over the 30 BFCL prompts), and how far each grade moved.

The chart scrolls sideways on a narrow screen.

Raw streamShown textTool calls

Qwen3-1.7B at 32k against the 6 GB rule

Resident memory (RSS) of the test process right after the 32k export loaded, on each phone, with the phone's RAM. The dashed line is the 6 GB threshold the killed runs' logs reported; that threshold is measured in PSS at kill time, a different measure from the bars, so the bars show which phones were near it, not the moment they crossed it. Three of four processes were stopped within the first dozen prompts.

The chart scrolls sideways on a narrow screen.

6 GB, the PSS threshold Samsung Heimdall and MIUI reported when they stopped the processThe phone's RAM

Time to first token and time per output token

Every rerun recorded prefill and decode time per prompt. TTFT here is the runtime's prefill of the whole prompt, the engine's part of the wait before the first token; TPOT is decode time over generated tokens. Each row pairs a rerun with the earlier run of the same cell: the hollow marker is the earlier run, the solid one the rerun. Medians over the prompts each run completed, so they are per-cell figures, not a paired speed test.

What to take from it: the capped rerun's lower TTFT is a prompt-mix effect, not a speed change. Its 60 GSM8K and IFEval prompts are shorter than the 30 BFCL prompts with their tool definitions, which the 640-cap rows include. TPOT, which does not depend on prompt length, agrees within 5% on the Dimensitys and Exynos 2400, differs by 8 to 11% on the 8 Elite and on the Tensor G5 at 4k, and by 26% on the Tensor G5 at 32k (81 then 102 ms), the size of the launch-to-launch swing seen on that phone before. Of the five repeats, three agree within 4% on both figures; the 1.2B repeats at 32k on the Dimensity and at 4k on the Exynos were 7 to 12% slower the second time.

The chart scrolls sideways on a narrow screen.

DimensitysSnapdragon 8 EliteTensor G5Exynos 2400Earlier runRerun

What the reruns showed

What the cut replies mean: A reply cut by the window exceeded the export's prompt-plus-reply length in this test; a reply cut by the cap is one the benchmark chose not to wait for. In every cell 10 to 15 of 60 replies were cut. At 2k on the Poco all 14 reached the window limit and none the cap; in the seven other cells, at 4k and 32k on every phone, 10 to 15 reached the 2048 cap and none the window. Some prompts make this model reason past 2000 tokens.
The kill rule: Samsung's Heimdall wrote "Trigger Global kill before GC, Usage 8.67 GB, Threshold 6 GB" on the S25 Ultra and S24+. MIUI on the Poco wrote "used too many pss resource, pss used 8653468kb, pss threshold 6291456kb". Different code, same number: on those phones a process whose PSS passes 6 GB is stopped. The 32k Qwen3 export needs about 7 GB of KV cache before the first prompt; it measured 6.1 to 6.4 GB resident after load on the three runs that were stopped and 8.5 GB on the Pixel run that finished.

The rerun tables

The rerun cells and controls. The full matrix, with the 640-cap rows and the fixed-prompt speed probe, is on the window page.

LFM2.5-2.6B under two reply caps8 cells, GSM8K and IFEval. Passed out of 30 at the 640-token cap and at the 2048-token rerun, and how the 60 rerun replies ended.Show

Phone	Window	GSM8K at 640	GSM8K at 2048	IFEval at 640	IFEval at 2048	Finished	Cut by window	Cut by cap
Dimensitys	2k	15/30	24/30	4/30	21/30	46	14	0
Dimensitys	32k	13/30	26/30	5/30	21/30	50	0	10
Snapdragon 8 Elite	4k	13/30	25/30	5/30	19/30	49	0	11
Snapdragon 8 Elite	32k	13/30	24/30	5/30	21/30	45	0	15
Tensor G5	4k	17/30	23/30	6/30	21/30	47	0	13
Tensor G5	32k	19/30	28/30	4/30	18/30	48	0	12
Exynos 2400	4k	10/30	25/30	5/30	21/30	47	0	13
Exynos 2400	32k	18/30	27/30	3/30	19/30	50	0	10

Same file, run twice5 controls. Replies byte-identical between the two runs, and each set's grade in the first and the second run.Show

Model	Phone	Window	Raw identical	Text identical	Calls identical	GSM8K	IFEval	BFCL
LFM2.5-1.2B	Dimensitys	16k	13/90	20/90	23/30	13 then 14	20 then 16	27 then 28
LFM2.5-1.2B	Dimensitys	32k	15/90	18/90	25/30	12 then 12	18 then 15	27 then 27
LFM2.5-2.6B	Dimensitys	2k	0/90	29/90	24/30	15 then 13	4 then 6	24 then 24
Qwen3-1.7B	Dimensitys	2k	31/90	31/90	30/30	20 then 17	15 then 18	24 then 24
LFM2.5-1.2B	Exynos 2400	4k	7/90	11/90	23/30	13 then 15	17 then 21	26 then 27

Timing of every rerun cellTTFT, TPOT, throughput and memory, against the earlier run of the same cellShow

Medians over the prompts each run completed. Prefill ms is the runtime's prefill of the whole prompt; ms per token is decode time over generated tokens. RSS is resident memory after load and at the end of the run.

Cell	Run	Prefill ms	ms per token	Prefill tok/s	Decode tok/s	RSS after load MB	RSS at end MB
LFM2.5-2.6B 2k, Dimensitys	640 cap, 90 prompts	882	60.3	108	16.6	1979	1958
	2048 cap, 60 prompts	693	62.8	105	15.9	1959	1783
LFM2.5-2.6B 4k, Snapdragon 8 Elite	640 cap, 90 prompts	1197	55.4	72	18	2058	2194
	2048 cap, 60 prompts	866	51.1	82	19.6	2055	2139
LFM2.5-2.6B 4k, Tensor G5	640 cap, 90 prompts	1066	92.2	88	10.8	2054	2221
	2048 cap, 60 prompts	776	81.9	106	12.2	2055	2188
LFM2.5-2.6B 4k, Exynos 2400	640 cap, 90 prompts	1142	83.5	81	12	2136	2189
	2048 cap, 60 prompts	893	87.1	84	11.5	2136	2187
LFM2.5-2.6B 32k, Dimensitys	640 cap, 90 prompts	882	59.7	109	16.8	2937	2185
	2048 cap, 60 prompts	621	59.5	113	16.8	2922	2262
LFM2.5-2.6B 32k, Snapdragon 8 Elite	640 cap, 90 prompts	1096	49.7	81	20.1	2950	3082
	2048 cap, 60 prompts	982	54.5	80	18.4	2950	2626
LFM2.5-2.6B 32k, Tensor G5	640 cap, 90 prompts	968	81.1	101	12.3	2962	3120
	2048 cap, 60 prompts	842	102.4	80	9.8	2962	3123
LFM2.5-2.6B 32k, Exynos 2400	640 cap, 90 prompts	1085	84.5	80	11.8	3032	3084
	2048 cap, 60 prompts	924	87.8	82	11.4	3028	3080
LFM2.5-1.2B 4k, Exynos 2400	first run	460	34	195	29.4	1126	1213
	second run	516	37.5	180	26.7	1126	1217
LFM2.5-1.2B 16k, Dimensitys	first run	354	25.9	262	38.6	-	993
	second run	368	26.2	259	38.2	1416	1528
LFM2.5-1.2B 32k, Dimensitys	first run	366	26.3	257	38	-	1903
	second run	392	28.8	237	34.8	1798	1901
LFM2.5-2.6B 2k, Dimensitys	first run	882	60.3	108	16.6	1979	1958
	second run	872	59.6	110	16.8	1984	1871
Qwen3-1.7B 2k, Dimensitys	first run	532	45.1	174	22.2	1758	1833
	second run	532	45.7	174	21.9	1766	1900

Qwen3-1.7B 32k on four phonesThe kill evidenceShow

Phone	Device	RAM	Resident after load	Prompts completed	What the log says
Dimensitys	Poco X8 Pro	12 GB	6188 MB	6 of 90	MIUI ActivityManager, 6 GB PSS threshold (rerun: 7 prompts, 8.65 GB)
Snapdragon 8 Elite	Galaxy S25 Ultra	12 GB	6127 MB	12 of 90	Samsung Heimdall, 6 GB threshold (8.67 GB used)
Exynos 2400	Galaxy S24+	12 GB	6396 MB	7 of 90	Samsung Heimdall, 6 GB threshold
Tensor G5	Pixel 10 Pro XL	16 GB	8499 MB	90 of 90	finished all 90

How we ran it

What was rerun and why: After the window matrix was published, every cell was checked for a failed or noisy run, and the three things the first pass left in doubt were rerun the same day: the 2.6B's grades, which its 640-token cap had cut short; the Poco's Qwen3 32k run, which had died without a recorded cause; and whether a second run of the same file on another phone varies as much as it did on the Poco.
Grading: The matrix's prompts and graders: GSM8K by the last number in the reply, IFEval by the benchmark's instruction checkers, BFCL by one call to the right function with the right arguments. Greedy, thinking off. Every run also recorded prefill and decode time per prompt and resident memory. The 2.6B rerun raised only the reply cap, from 640 to 2048, on its 60 GSM8K and IFEval prompts; the repeats used all 90.
Why the 2.6B needed a rerun: Its chat template opens a thinking block before every answer, and the suite runs thinking off with a 640-token cap. Most of its GSM8K and IFEval replies were still reasoning when generation reached the cap, so those grades confound answer quality with the 640-token limit. The rerun keeps the same prompts, graders and thinking-off rule and only raises the cap to 2048. The 640-cap rows stay in the matrix as their own condition.
Cost: One day. The cloud reruns took 22 Test Lab launches, one set per launch and continued where the 38-minute budget cut them; 20 ran and 2 were refused when the Blaze plan's quota of 50 physical-device tests per day ran out at 13:19. They resumed at 15:05 after the reset and finished at 18:51. The Poco ran its four steps in 3 h 20 min.
Where the numbers come from: Grades, controls and memory figures are the rendered matrix's. The cut reasons and the reply-length histogram come from the graded JSON reports, which record prompt and reply token counts per prompt; a reply is counted as cut by the window when prompt plus reply reached the window's last usable token, 2,047 at 2k, and as cut by the cap when the reply alone reached 2,047 tokens at the 2048 cap. Phone RAM is the manufacturer's figure for each device. Launch times and counts are from the run logs.

How long each run tookWall clock per run. Generation time alone for the 2.6B at 2048: 130 to 213 minutes per phone.Show

The Test Lab span includes a pause from 13:19 to 15:05 for the daily quota.

Run	Where	Wall clock
Qwen3 32k with a log	Poco	2 min, killed
2.6B 2k repeat, 90 prompts	Poco	48 min
2.6B 2k at 2048, 60 prompts	Poco	80 min
2.6B 32k at 2048, 60 prompts	Poco	66 min
2.6B 4k and 32k at 2048, 1.2B 4k repeat	Test Lab, 3 phones	22 launches, 12:40 to 18:51

TermsWhat a cell, an export, greedy, TTFT, TPOT, RSS and PSS mean hereShow

Cell: One export on one phone over the prompt set.
Export: The model compiled for ExecuTorch at a fixed context window.
Greedy: The most likely token is always taken, so a run should repeat exactly.
Thinking off: The model is not given room to reason before answering, though the 2.6B's template does so anyway.
Benchmarks: GSM8K is grade-school maths, IFEval instruction following, BFCL tool calling.
Identity: Raw stream is every token generated, shown text is what a user would see after tool-call markup is removed, parsed tool calls are the function calls the app would act on.
TTFT and TPOT: TTFT is time to first token, measured here as the runtime's prefill of the prompt; TPOT is time per output token, decode time over generated tokens.
RSS and PSS: RSS is the process's resident memory; PSS is its proportional share, the figure the memory guards read.
KV cache: The memory a model keeps per token of window.
Heimdall: Samsung's memory guard.
Test Lab: Firebase's rented-phone service.

Limits

What was not rerun: The cloud phones' suite timings, which differed by up to threefold between two launches of the same Tensor G5 cell and may reflect different thermal states. A repeat would be as launch-dependent; a paired probe on those phones is required to compare their speed.

Sources

Under the results folder, the capped rerun is in -cap2048 files, continued sets are @n, second runs have a repeat- prefix, and the Poco kill evidence is under qwen3-32k-poco-rerun/. The LFM2.5-2.6B and LFM2.5-1.2B exports were also published on Hugging Face.

The note Raw and graded reports Qwen3-1.7B exports on Hugging Face Source on GitHub

Keep reading

Over 6 GB
for Qwen3 at 32k, before the first prompt
- LFM2.5 1.2B, 32k1.8 GB
- Qwen3 1.7B, 2k1.8 GB
- Qwen3 1.7B, 32k6.2 GB
OpenWeights, 7 September 2026
Does the exported window matter?
Whether the context window fixed at export, from 2k to 32k, changes anything but memory.
18 of 25
model and chip pairs reached the first token sooner on ExecuTorch
OpenWeights, 3 and 4 September 2026
Five chips, two runtimes
How fast five small models answer on five phone chips, in llama.cpp and in ExecuTorch.
9 of 9
models were fastest with NPU prefill
OpenWeights, 2 to 4 November 2026
Prefill and decode on CPU, GPU and NPU
Which processor should prefill and which should decode: nine models, every pairing, on a Snapdragon 8 Elite Gen 5.