Skip to content

ExecuServeOn Google Play

ExecuTorch models, served from your phone.

ExecuServe keeps compiled ExecuTorch models loaded on an Android phone and answers the OpenAI and Anthropic APIs, for any app on the phone or, if you allow it, on your network.

Android 12+ on arm64 · Apache-2.0

The browser chat served by the phone: a reply from Qwen3 1.7B running on the device, with its prefill and decode rates
The browser chat, served by the phone and answering from Qwen3 1.7B running on it.

What it does

A server, not a chat app
Your agents, SDKs and tools call the phone the way they call a cloud.
The KV cache belongs to the server
Clients resend the conversation, and the server reuses its cache, so follow-ups stay fast.
Several models, one phone
Up to three resident, each with its own cache and endpoints.
Built to stay up
A foreground service, tested through deep Doze with the screen off.
A browser chat inside
A laptop on your network can talk to the phone with nothing installed.
CPU, GPU and NPU
The catalog offers each phone only the files its chip can load.
Private by construction
No account, no analytics, no crash reporter. Every request needs a key, loopback included.

Measured on a phone

On a POCO X8 Pro Max (Dimensity 9500s), over Wi-Fi, with the minified release build.

CheckResult
Official OpenAI SDK suite16 of 16 pass
Official Anthropic SDK suite8 of 8 pass
Edge-case probe (refusals, limits, disconnects)28 of 28 pass
Screen-off soak on battery, 10 minutes20 of 20 answered
Warm second turn of a 2,000-token conversation0.47 s instead of 8.4 s
Qwen3 tool loop, second request174 of 205 prompt tokens reused
XNNPACK export against llama.cpp, Snapdragon 8 Elite1.2 to 1.6× faster decode
Qwen3-1.7B, 700-token prompt, Snapdragon 8 Elite Gen 5: first token on the NPU0.41 s, against 2.0 s on the CPU and 1.0 s on the GPU
Qwen3-1.7B decode on the same phone, CPU / GPU / NPU54 / 42 / 25 tokens per second
MediaTek NPU, LFM2.5-1.2B, DimensityNPU prefill, CPU decode at 39 to 43 tokens per second

Method and raw results: the POCO X8 Pro Max report. The decode advantage of the XNNPACK export over llama.cpp was measured in OpenWeights, on whose ExecuTorch engine ExecuServe is built.

Quick start

An arm64 phone on Android 12 or later, adb on your computer, and an ExecuTorch export with its tokenizer.

API

From a clone of the repository, with the app installed:

tools/execuserve --model ~/models/Qwen3-1.7B-8da4w-gptq-2k.pte
export OPENAI_BASE_URL=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=<the key it prints>

Then any OpenAI client:

from openai import OpenAI

client = OpenAI()  # reads OPENAI_BASE_URL and OPENAI_API_KEY
reply = client.chat.completions.create(
    model="qwen3-1.7b",
    messages=[{"role": "user", "content": "Hello"}],
)

The script pushes the model, starts the server, forwards the port over adb, and prints the base URL and key.

Reference

The API

OpenAI's and Anthropic's shapes, error codes and streaming, checked with their official SDKs.

RouteWhat it does
POST /v1/chat/completionsStreaming, tools and tool calls, reasoning, usage with cached tokens
POST /v1/responsesItems, function calls, typed stream events, previous_response_id
POST /v1/messagesAnthropic's Messages API: tool_use, thinking blocks, its stream events
POST /v1/completionsA raw prompt, no template
GET /v1/modelsInstalled models with context length, residency and capabilities
GET /metricsPrometheus counters, gauges and quantiles
/models/{id}/v1The model, inference and status routes scoped to one model, for hosting several at once

Clients

ClientSetting
OpenAI SDKs, LangChain, Vercel AI SDKBase URL http://<phone>:8080/v1, plus the key
Anthropic SDKs and agents built on themBase URL http://<phone>:8080, with no /v1, plus the key
Open WebUISettings, Connections, OpenAI API, with the URL http://<phone>:8080/v1
codex CLIA provider with wire_api = "responses"
Another app on the same phonehttp://127.0.0.1:8080/v1, with a network security config allowing loopback

Get a model onto the phone

WayHow
In the appLibrary, then Browse the catalog, which lists the exports this phone can run
From your computertools/execuserve --model path/to/Model.pte pushes it with its tokenizer and starts serving
Have the phone fetch ittools/execuserve pull <hf-repo> <file.pte> downloads and verifies it on the phone

Documentation

READMEEverything the server does, and how to build it
ArchitectureThe design, the HTTP API in full, and edge cases
Testing on a phoneFrom installing to a screen-off soak
ResultsMeasurements on real phones, with dates
Privacy policyWhat stays on the phone and what leaves
ContributingSetting up, the checks, what a change needs
SecurityReporting a vulnerability, and what is in scope

Open source

Apache-2.0, on GitHub. Next are the iOS app, server-side tools, vision input, and TLS for network mode. Not affiliated with, endorsed by or sponsored by the PyTorch Foundation or Meta.