8 agents

at a 4.9 s wait per turn on vLLM, with no errors. At 16, waits reach 16.5 s

Hardware, 23 and 24 September 2026

How many coding agents can one RTX 4090 serve?

Qwen3.8-27B on one rented RTX 4090, through five engine setups, rising load and real coding agents.

What we found

Eight agents fit on one card
vLLM served 8 at once at a 4.9 s median wait per turn, with no errors. At 16 there were still no errors, but the wait reached 16.5 s and each agent got 6.4 tok/s.
vLLM held up best under load
At 16 users it reached 368 tok/s in total with first tokens near 2 s, where the next best reached 141. It queues extra requests instead of failing them.
Each other engine hit a different wall
Ollama is the fastest for one request at 102 tok/s, but runs this model one at a time, so first tokens took 49 s at 16 users. SGLang had the shortest median wait up to about 4 agents, then 18.7 s at 8. llama.cpp failed 8 of 54 requests at 8 agents.
gpt-oss-20b was far quicker on the same card
With the same size of cache pool, gpt-oss-20b waited 0.26 s per turn at 8 agents where Qwen3.8 waited 4.9 s. The cause is not isolated, since replies and cache hits also differed, and their quality was not compared.
Speculative decoding helps one person chatting and hurts a team
On vLLM it lifted one user's decode from 52.5 to 82.3 tok/s. One agent's session stayed level, 36 s against 37 s, and 8 agents' sessions went from 101 s to 236 s.

The results

How long an agent waits for each turn

Time to first token per turn, for pi's real system prompt and four tools, over 8 turns. At one agent every engine is close. By eight they are far apart.

The setup we recommendThe othersSolid to the median, faint to p90

1 agent at once

Ollama1 s, 1.1 s at p90
llama.cpp0.8 s, 1 s at p90
SGLang0.6 s, 0.7 s at p90
vLLM fp80.7 s, 0.9 s at p90
vLLM int4 pinned0.8 s, 0.9 s at p90

4 agents at once

Ollama6.4 s, 13.4 s at p90
llama.cpp6 s, 7.9 s at p90
SGLang1.7 s, 5.6 s at p90
vLLM fp81.9 s, 2.8 s at p90
vLLM int4 pinned2.3 s, 3.5 s at p90

8 agents at once

Ollama22.5 s, 38 s at p90
llama.cpp10.7 s, 14.1 s at p90 with failures
SGLang18.7 s, 29.6 s at p90
vLLM fp87.4 s, 15.7 s at p90
vLLM int4 pinned4.9 s, 7.2 s at p90

llama.cpp failed 8 of 54 requests at 8 agents: 5 rejected for exceeding the context and 3 cut off mid-stream. Ollama's replies were shorter, so only its waits compare. vLLM fp8 ran on host 1 and vLLM int4 pinned on host 2.

Total output as users are added

A 1k prompt and 256 tokens out. Ollama stays flat at about 78 tok/s because it serves this model one request at a time. SGLang stops rising once its 4 running slots are full.

Ollamallama.cppSGLangvLLM fp8vLLM int4 pinned

The wait for a first token as users are added

The same runs. vLLM stays near 2 s at 16 users, and the engines that queue climb quickly.

Ollamallama.cppSGLangvLLM fp8vLLM int4 pinned

What more agents cost on the recommended setup

No errors at any load. Eight is comfortable, twelve is usable and slowing, and sixteen is too slow to use.

Agents at onceWait per turn, median and p90tok/s per agentInput served from cache

1 agent
Wait per turn, median and p900.8 s, 0.9 s
tok/s per agent50.1
Input served from cache77%
2 agents
Wait per turn, median and p901.3 s, 1.5 s
tok/s per agent45.9
Input served from cache76%
4 agents
Wait per turn, median and p902.3 s, 3.5 s
tok/s per agent41
Input served from cache70%
8 agents
Wait per turn, median and p904.9 s, 7.2 s
tok/s per agent30.1
Input served from cache69%
12 agents
Wait per turn, median and p907.4 s, 14.6 s
tok/s per agent20.6
Input served from cache54%
16 agents
Wait per turn, median and p9016.5 s, 29.6 s
tok/s per agent6.4
Input served from cache11%

vLLM 0.30 with int4 KV and a pinned 246,094-token pool. Up to 13 requests waited at once under 12 to 16 agents.

Among the int4 pools, size showed at twelve agents

On this agent workload, the three int4 pools waited the same at 8 agents. At 12, a bigger pool served more from cache and waited less.

vLLM setupPool, tokensWait at 8 agentsWait at 12, median and p90From cache at 12

fp8, 0.95, host 1
Pool, tokens84,898
Wait at 8 agents7.4 s
Wait at 12, median and p90not run
From cache at 12not run
int4, 0.97, host 1
Pool, tokens180,558
Wait at 8 agents5 s
Wait at 12, median and p9011.7 s, 29.9 s
From cache at 1212%
int4, pinned 4.5 GiB, host 2
Pool, tokens246,094
Wait at 8 agents4.9 s
Wait at 12, median and p907.4 s, 14.6 s
From cache at 1254%
int4, 0.97, host 2
Pool, tokens298,255
Wait at 8 agents5.1 s
Wait at 12, median and p906.7 s, 11.1 s
From cache at 1269%

The likely cause was not isolated: a cache miss means re-reading the whole context of about 12k tokens.

Two models on the same card

Both on vLLM with the same size of pool. Each row has its own scale. A probable contributor is 3.6B active parameters against 27B, not isolated.

Qwen3.8-27Bgpt-oss-20b

Agent turn wait at 8 agents
Median. Shorter is better
Qwen3.8 4.9 s
gpt-oss 0.26 s
Agent turn wait at 16 agents
Median. Shorter is better
Qwen3.8 16.5 s
gpt-oss 0.88 s
Prefill, 16k prompt
tok/s
Qwen3.8 2,199
gpt-oss 13,196
Decode, 1 user
tok/s
Qwen3.8 52.5
gpt-oss 197.6
Decode, 16 at once
Total tok/s
Qwen3.8 357
gpt-oss 1,019

gpt-oss-20b's agent replies stopped naturally at 64 to 74 tokens instead of a forced 150, so compare turn waits, not session times.

Speculative decoding on vLLM

MTP with 2 draft tokens takes memory from the cache pool, 108,814 tokens against 180,558. One person chatting gains, one agent breaks even, and a team loses.

Without MTPWith MTP

Decode, 1 user
tok/s. Longer is better
Without 52.5
With 82.3
Session time, 1 agent
Shorter is better
Without 36 s
With 37 s
Session time, 8 agents
Shorter is better
Without 101 s
With 236 s
First token, 16 at once
Shorter is better
Without 1.8 s
With 6.7 s
Input served from cache, 1 agent
Longer is better
Without 77%
With 51%

Every numberPrefill, decode under load, gpt-oss-20b on every engine, how long context stays cached, and the real pi runsShow

Prefill of one request, tok/s	Ollama	llama.cpp	SGLang	vLLM fp8	vLLM int4 pinned
1k	1,226	1,770	2,965	2,814	2,809
8k	2,300	2,527	2,902	2,888	2,492
32k	2,263, see below	2,434	2,588	2,568	1,768

Ollama's context was 32,768 tokens. Of its three 32k prompts, the one over that limit, 35.2k, was silently cut to 16,386 tokens. int4 KV is slower on long cold prompts: on host 2, a 32k prompt took 18.4 s to first token with int4 KV and 15.1 s with fp8 KV.

Decode at once: per user, total, first token	Ollama	llama.cpp	SGLang	vLLM fp8	vLLM int4 pinned
1	102.2, 78, 0.8 s	46.7, 42, 0.6 s	53.5, 50, 0.3 s	53.7, 50, 0.3 s	52.5, 49, 0.3 s
4	101.8, 78, 10.5 s	36.5, 109, 2.5 s	45.1, 116, 1.1 s	48.5, 156, 1.3 s	45.9, 152, 1.2 s
8	102.6, 78, 23.3 s	23.1, 142, 3.4 s	47.8, 121, 10.4 s	41.8, 255, 1.8 s	36.7, 248, 1.3 s
16	102.4, 78, 49.4 s	22, 141, 16.8 s	46, 122, 26.1 s	28.9, 368, 2.2 s	26.5, 357, 1.8 s

gpt-oss-20b	vLLM	llama.cpp	Ollama	SGLang
Prefill 1k, 8k and 32k, tok/s	15,624, 15,510, 10,069	7,274, 11,198, 10,526	5,958, 10,205, 9,977	2,642, 4,949, 4,938
Decode, 1 user, tok/s	197.6	242.6	191.8	24.3
16 at once: per user, total, first token	74.7, 1,019, 0.6 s	77.4, 484, 5.1 s	100.6, 336, 9.2 s	8.0, 118, 2.9 s
Agent turn wait p50 and p90, 1 agent	0.14 and 0.15 s	0.21 and 0.22 s	0.25 and 0.31 s	0.46 and 0.48 s
Agent turn wait p50 and p90, 8 agents	0.26 and 0.48 s	1.64 and 3.10 s	1.40 and 2.02 s	1.06 and 1.50 s
Errors	0	0	0	0

All on host 2 with their own setups: vLLM 0.30.0 with bf16 KV pinned at 6 GiB, llama.cpp b11118 with ggml-org's MXFP4 GGUF, 8 slots and a 128k shared pool, Ollama 0.34.4 with 4 parallel requests, and SGLang 0.5.20 with bf16 KV and 16 running. Ollama's 34.1k prompt was silently cut to 16,386 tokens, so its 32k prefill is not of an intact prompt.

Idle gap	Quiet server	4 other agents	8 other agents
5 s	93%	93%	93%
30 s	not run	94%	0%
1 min	94%	97%	0%
2 min	not run	0%	0%
4 min	not run	0%	0%
5 min	97%	not run	not run
8 min	not run	invalid	0%
15 min	88%	not run	not run

Share of a context of about 12k tokens still cached after each gap, each gap with its own context. After an eviction, that turn took 5.7 to 21 s to start. Only one pool size was tested.

Server	pi thinking	Agents at once	Passed	Wall time
vLLM int4, host 1	off	1, 4 and 8	13 of 13	24.1, 26.0 and 88.4 s
vLLM int4, host 1	medium	1, 4 and 8	13 of 13	23.6, 58.5 and 70.3 s
vLLM int4 pinned, host 2	medium	8	8 of 8	96.1 s

34 of 34 runs passed, and no agent edited the test files. Tasks were small, 4 to 15 tool calls, on small repositories with failing tests.

How we ran it

Question: Can one rented RTX 4090 serve Qwen3.8-27B to a team using the pi coding agent, and with which engine and quantization?
Machine: One RTX 4090, 24,564 MiB, Ryzen 9 7950X, 61 GB, on RunPod EU-RO-1, on two hosts with the same volume: host 1 on 2026-09-23 and host 2 on 2026-09-24.
Engines: Ollama 0.34.3 with its Q4_K_M build and speculative decoding on by default. llama.cpp b11118 with UD-Q4_K_XL, 8 slots and a 64k shared pool. SGLang 0.5.19 with INT4, fp8 KV and 4 running requests. vLLM 0.30.0 with INT4, once with fp8 KV and an 84,898-token pool, once with int4 KV and a pinned 246,094-token pool.
Load: A load generator on the pod, so no network is in the numbers, with real Python source, unique per request. The agent scenario uses pi's real system prompt and 4 tools, for 8 turns of about 1.5k new tokens in and 150 out, growing to about 12.6k tokens.
Real agents: Real pi agents on a Mac, up to 8 at once, 34 of 34 passing.
The setup we recommend: vLLM 0.30.0, RedHatAI's INT4 weights at 16.59 GiB on the GPU, vision tower off, int4 KV pinned at 4.5 GiB, recurrent state in fp16, prefix caching on and speculative decoding off.
Review: Codex reviewed the method and report over 11 rounds. The notes are in the repository.

Limits

Two hosts: The recommended setup and vLLM fp8 ran on different hosts, and fp8 KV prefilled 17 to 18% slower on host 2 than host 1, with no explanation found. Do not read small differences between setups as real.
Not one file: The engines ran different quantizations of the model, and output quality was not measured on our own tasks.
Small tasks: The real pi runs had 4 to 15 tool calls, so they show tool calls and concurrency work, not quality on real work.
Longer sessions: Real pi sessions went no longer than about 12.6k tokens. Sessions near 30k should lower the ceiling to about 4 to 5 agents, which is a projection and not a measurement.
No fairness: vLLM serves first come, first served. One user running 8 agents slowed everyone else, and the light users were not protected.
One architecture: Which engine wins depends on the model's architecture and weight format. On gpt-oss-20b, llama.cpp had the fastest single stream. Test a new model before choosing.
Versions: Engines change quickly, and these are the versions of September 2026.

Sources

The full report Every config, in tables Raw results Review notes The local Copilot study