PyTorch Day Korea
1 in 5
replies from the compiled 1.2B cited a search it never made, on our first test set. As a GGUF, none of 640
Seoul, 21 November 2026
The Search It Never Made: When a 1.2B On-Device Model Should Call Web Search.
Whether a small model on a phone searches when it must, on 160 public questions, four phones and two builds of the same weights.
The talk
In a model-driven agent loop, a small model on a phone decides whether to search the web or answer from its weights. A wrong call costs either a fabricated fact or several seconds and a network round trip.
The talk puts that decision to 160 public questions from RetrievalQA, PopQA and FreshQA, through a shipping open source Android assistant on four phones, with the same LFM2.5 1.2B weights as a compiled ExecuTorch export and as a GGUF. The two builds did not decide alike.
The evidence
The same weights, two builds
| LFM2.5 1.2B, web search the only tool | Compiled, 8da4w ExecuTorch | GGUF, Q4_K_M llama.cpp |
|---|---|---|
| Searched on the 75 questions that need it | 1 to 3 of 75 | 33 or 34 of 75 |
| Cited a search it never made | 32 to 35 of 160 | 0 of 160, on each of four phones |
160 rows an arm, seed 7, on the Dimensitys, Tensor G5, Exynos 2400 and Snapdragon 8 Elite, with web search the only tool offered. Correct answers are left out here: their grading was later revised, and the notes below carry both versions.
- A standing note
- These are not clean measures of the model's own judgment. On 21 of the 75 questions, the app's prompt note asks for a lookup. The GGUF obeyed it on all 21, which is two thirds of its 33 calls, and called on 12 of the other 54. The compiled model obeyed it on none.
- A second test set
- On a second set of rows, on one phone, the GGUF did narrate a search it had not made, twice in 160 replies in each arm. So its zero holds for the first set only.
- A later correction
- The follow-up note on why the compiled model narrates is corrected too: ExecuTorch's LFM2 graph carried state between prompts in a reused runner.
Where and when
- Conference
- PyTorch Day Korea, hosted by the PyTorch Korea User Group
- Date
- Saturday 21 November 2026. The session's time is not published yet.
- Room
- Centerfield East 18F, AWS Korea, 231 Teheran-ro, Gangnam-gu, Seoul
- Track
- Run
- Speaker
- Xynil Jhed Lacap