42%
of 24 real tasks with Qwen3.6 on an RTX 4090, against 92% on Copilot Pro+ Auto
GGUF, October 2026
When is local Copilot worth it?
The same Copilot CLI agent on the same real tasks: open-weight GGUF models on an RTX 4090 and a MacBook Air M5, against hosted Copilot.
What we found
Hosted finished more: 92% against 42%
Qwen3.6 on an RTX 4090 against Copilot Pro+ on Auto was the primary contrast, fixed before October scoring. On 24 paired tasks the gap is −50 points, 95% interval −71 to −29, permutation p = 0.0005.
Gemma 4 on the 4090 came closest
It finished 58%. Its interval clears the 10-point margin, but the Holm-adjusted p is 0.108, so no difference is declared. Every other local configuration finished 29% to 42%.
Fast is not the same as finished
gpt-oss-20b on the 4090 had the shortest median run, 31.7 s against 57.9 s for Pro+ on Auto, and passed 10 of 24. Qwen3.6 on the 4090 had a median of 600 s, the cap: 13 of its 24 runs reached it.
The agent moved Qwen3.6 most
With Copilot CLI it often called the CLI's read_agent tool with invented agent IDs until the cap. Exploratory check on the same file and box: 12 of 24 on Copilot CLI, 18 of 24 on pi. Several things change at once between the two.
Local was not always cheaper per finished task
Qwen3.6 on the rented 4090 cost $0.218 per completed task and Pro+ on Auto $0.169. gpt-oss-20b cost $0.022 and Gemma 4 $0.098. Electricity on the MacBook Air was not measured.
The results
The ten setups the study registered
Hosted setups finished more, in about a minute. Quick did not mean finished: gpt-oss-20b on the 4090 had the shortest median run and finished 10 of 24.
Hosted CopilotServed locally
Copilot Free (education), Auto
Finished24 of 24Median run55 sPer task$0.011Copilot Pro+, top model pinned
Finished24 of 24Median run63 sPer task$0.582Copilot Pro, Auto
Finished23 of 24Median run63 sPer task$0.148Copilot Pro+, Auto
Finished22 of 24Median run58 sPer task$0.169Gemma 4 on the RTX 4090
Finished14 of 24Median run50 sPer task$0.098Qwen3.6 on the RTX 4090
Finished10 of 24Median run600 sPer task$0.218gpt-oss-20b on the RTX 4090
Finished10 of 24Median run32 sPer task$0.022Gemma 4 on the MacBook Air
Finished10 of 24Median run481 sPer tasknot measuredgpt-oss-20b on the MacBook Air
Finished7 of 24Median run227 sPer tasknot measuredQwen3.6 on the MacBook Air
Finished7 of 24Median run600 sPer tasknot measured
Finished: the task's checks pass on the agent's checkout. Runs are capped at 600 s. Cost: agent time at $0.89 an hour on the rented 4090, idle and loading excluded; credits at $0.01 on hosted plans, leaving out each plan's monthly price. Electricity on the MacBook Air was not measured.
How far each setup sits from Copilot Pro+ on Auto
Gap in tasks finished, in percentage points, with its 95% interval. Left of zero finished less than Pro+ on Auto. The shaded band is the study's 10-point margin.
Hosted CopilotServed locally
Qwen3.6 on the RTX 4090
−50 primary, Pro+ Auto ahead
gpt-oss-20b on the RTX 4090
−50 declared
Gemma 4 on the RTX 4090
−33 not declared after Holm
gpt-oss-20b on the MacBook Air
−62 declared
Qwen3.6 on the MacBook Air
−62 declared
Gemma 4 on the MacBook Air
−50 declared
Copilot Pro, Auto
+4 no difference declared
Copilot Free (education), Auto
+8 no difference declared
Copilot Pro+, top model pinned
+8 no difference declared
Declared means the study declares a difference after the Holm correction across the secondary contrasts. Gemma 4 on the RTX 4090 clears the margin but its corrected p is 0.108, so no difference is declared.
Where the wait goes
Share of each setup's agent time. On the rented 4090, a third or more goes to the network and proxy. On the Air, nearly all of it goes to prefill and decode. Hosted model time is one number.
PrefillDecodeNetwork and proxyTools and CLIModel, hosted and not split
Qwen3.6 on the RTX 4090
prefill 8%, decode 32%, network and proxy 57%, tools and CLI 3%
gpt-oss-20b on the RTX 4090
prefill 7%, decode 38%, network and proxy 37%, tools and CLI 19%
Gemma 4 on the RTX 4090
prefill 5%, decode 51%, network and proxy 34%, tools and CLI 10%
Qwen3.6 on the MacBook Air
prefill 23%, decode 71%, network and proxy 1%, tools and CLI 5%
gpt-oss-20b on the MacBook Air
prefill 27%, decode 71%, network and proxy 1%, tools and CLI 2%
Gemma 4 on the MacBook Air
prefill 32%, decode 64%, network and proxy 1%, tools and CLI 2%
Copilot Pro+, Auto
model, hosted and not split 72%, tools and CLI 28%
Copilot Pro, Auto
model, hosted and not split 72%, tools and CLI 28%
Copilot Free (education), Auto
model, hosted and not split 71%, tools and CLI 29%
Copilot Pro+, top model pinned
model, hosted and not split 74%, tools and CLI 26%
Same models, two agents
Checked after the study on the same GGUF files, llama.cpp build, server flags, tasks and checks, both agents on the same 4090. Tasks finished of 24, with the median run. Exploratory, and outside every contrast.
Copilot CLIpi
gpt-oss-20b
Copilot CLI 7 of 24, 0.5 minpi 10 of 24, 0.2 minQwen3.6-35B-A3B
Copilot CLI 12 of 24, 5.7 minpi 18 of 24, 0.7 minGemma 4 26B-A4B
Copilot CLI 13 of 24, 0.6 minpi 13 of 24, 0.5 minQwen3.8-27B
Copilot CLI 17 of 24, 1.3 minpi 22 of 24, 1.2 minMellum 2.1 12B, thinking
Copilot CLI 11 of 24, 1.9 minpi 8 of 24, 0.6 min
Every numberThe ten setups and the two-agent check as tablesShowHide
| Configuration | Passed of 24 | Share | Median run | Per completed task |
|---|---|---|---|---|
| Copilot Free (education), Auto | 24 | 100% | 54.7 s | $0.011 |
| Copilot Pro+, top model pinned | 24 | 100% | 62.7 s | $0.582 |
| Copilot Pro, Auto | 23 | 96% | 63.2 s | $0.148 |
| Copilot Pro+, Auto | 22 | 92% | 57.9 s | $0.169 |
| Gemma 4 on the RTX 4090 | 14 | 58% | 50.2 s | $0.098 |
| Qwen3.6 on the RTX 4090 | 10 | 42% | 600.0 s | $0.218 |
| gpt-oss-20b on the RTX 4090 | 10 | 42% | 31.7 s | $0.022 |
| Gemma 4 on the MacBook Air | 10 | 42% | 480.6 s | Not measured |
| gpt-oss-20b on the MacBook Air | 7 | 29% | 227.3 s | Not measured |
| Qwen3.6 on the MacBook Air | 7 | 29% | 600.1 s | Not measured |
| Model | Copilot CLI | pi |
|---|---|---|
| gpt-oss-20b | 7 of 24, 0.5 min | 10 of 24, 0.2 min |
| Qwen3.6-35B-A3B | 12 of 24, 5.7 min | 18 of 24, 0.7 min |
| Gemma 4 26B-A4B | 13 of 24, 0.6 min | 13 of 24, 0.5 min |
| Qwen3.8-27B | 17 of 24, 1.3 min | 22 of 24, 1.2 min |
| Mellum 2.1 12B, thinking | 11 of 24, 1.9 min | 8 of 24, 0.6 min |
How we ran it
- Tasks
- 24 real pending maintenance tasks on three public repositories, each frozen at a commit, with an acceptance check anyone can run.
- Agent
- Copilot CLI, the same binary and flags on every configuration. Every run starts from a fresh clone and is capped at 600 s: the waiting budget.
- Scoring
- The task's own acceptance test, the repository's existing tests and a tamper check, run outside the agent's copy.
- Local
- llama-server from llama.cpp at commit c1d0e7a, on a rented RTX 4090 and on a 24 GB MacBook Air M5.
- Runs
- 754 scored runs: 240 in the ten registered configurations and 514 in exploratory ones. The core rules were committed before October scoring. Later changes are logged in the protocol, and a few came after the outcomes they relate to, including the choice of the 24-task analysis set; they are marked there.
- Statistics
- Task-paired bootstrap and permutation for the primary contrast, with the Holm correction on the secondary ones.
Limits
- No buying rule
- 24 tasks from three repositories chosen by the author support no claim of equivalence, plan ranking or general buying rule.
- Whole configurations
- Model file and quantization, runtime, memory pressure, client OS, tools, credit budget and Auto routing all differ. The results compare configurations and do not attribute a gap to hardware alone.
- One run each
- The registered analysis is one run per task and configuration, at temperature 0 with no fixed seed, so its interval is across tasks and does not establish repeatability. In the exploratory co-located setup, a second Copilot CLI run of Qwen3.6 moved from 12 to 9 of 24.
- A rented 4090
- It was reached over the network. For Qwen3.6, 57% of the summed time went to the network, proxy and HTTP, a median of 284.6 s a run. The study estimates a desk 4090 at 176.4 s a run, assuming the agent acts the same, which is not evidence that its capped runs would have finished.
- Reasoning off
- Qwen3.6 and Gemma 4 ran with reasoning off and gpt-oss-20b at low effort, while the hosted models ran as GitHub serves them. That may understate the open models: Mellum passed 4 of 24 with reasoning off and 11 of 24 with it on.
- The MacBook Air
- It paged memory to disk in most of its runs, with 1.6 to 10.1 GB of swap in use, and 13 of the 24 Qwen3.6 runs and 23 of the 24 Gemma 4 runs ran on battery. What that changed is not measured. It also ran Qwen3.6 and Gemma 4 at smaller quantisations than the 4090; only gpt-oss-20b is the same file on both.
- One harness
- Copilot CLI is the only harness in the registered design, so results are model-plus-harness outcomes. One model request at a time; several agents at once are not measured.
- Isolation
- It lowers, and does not rule out, leakage. Hosted runs kept web and GitHub tools, though no fetching command was detected in their traces. Hosted Auto routing cannot be reproduced; the model each run used is in its trace.
The results document opens with the full list of caveats. Read it before quoting these numbers.
Sources
The measurement behind GitHub Universe 2026 session BLD1241D, “When is local Copilot worth it? Three open-weight coders vs. Copilot out of the box”. Every run's trace, patch, check result and timing row is committed.