Running a 27B coding model on my laptop
Qwen3.8-27B on an M4 Max: 20.8 tokens a second, a 20,000-token floor before any work starts, and the defaults that quietly broke it.
I wanted a coding model that costs nothing per token, for the mechanical half of agent work: renames, formatting, running tests and reading the output. The machine is a MacBook Pro with an M4 Max, a 32-core GPU and 36 GB of unified memory, of which about 30 GB can be wired for a model. The software is LM Studio on its MLX engine.
The model is Qwen3.8-27B in 4-bit MLX: 16 GB on disk, about 15 GB once loaded, a context window of up to 262,144 tokens, and a mix of attention types (48 linear layers, 16 full). It’s built for tool use, which matters more here than raw benchmark scores.
The numbers
| Decode | 20.8 tokens/s |
| Prefill, 15,750-token prompt | 138 tokens/s |
| First turn, cold | about 130 s |
| Later turns, prefix cached | 8–12 s |
| Two-turn “write this file” task | 288 s cold, 70 s warm |
Decode speed is fine for reading along. Prefill is the problem, and it’s a problem because of a number I hadn’t thought about.
The 20,000-token floor
Claude Code’s system prompt and tool schemas come to about 20,000 input tokens on every turn, before the task says anything. At 138 tokens a second that’s two and a half minutes of prefill on a cold cache. Once the prefix is cached a turn drops to 8–12 seconds, so whether this setup is usable comes down almost entirely to whether the cache holds.
LM Studio serves Anthropic’s /v1/messages API natively, so Claude Code can talk to it directly with nothing in between. A small wrapper sets the endpoint and model, and a two-turn task worked end to end.
Defaults that broke it
Every one of these was a default, and every one failed quietly.
- Four parallel sessions. LM Studio reserved four KV caches for concurrent requests that never came. One session is enough for one agent.
- A 32k context. With 20k gone to the prompt, 32k left about 12k for the actual work. 64k costs about 4 GB more memory and is the minimum that makes sense.
- A 60-second client timeout. Claude Code gives up on an API call after 60 seconds by default, and a cold prefill takes more than 130. The server log had 81 “client disconnected” stops in the middle of processing a prompt. Setting
API_TIMEOUT_MS=600000fixed it. - A 4 GB prefix cache. It evicted 57 GB over one session, and every eviction meant re-reading the whole 20k prompt.
- No KV-cache quantisation. It would have shrunk the cache, but the engine refused: its batched vision path doesn’t support it yet.
- No 404 for a missing model. My gateway still pointed at a model I’d deleted. LM Studio didn’t refuse the request; it served whatever model was loaded. Nothing told me until I went looking.
A newer MLX engine was about 5% slower at decode, so I’ve pinned the older one. Engines move fast enough that it’s worth benchmarking each update rather than assuming.
What it’s good for
Free, private and slow. In the hybrid gateway that routes my Claude Code sessions, the local tier is the “haiku” slot: bounded, mechanical jobs where nobody is waiting on the answer. It has taken 18 of 133 logged requests, with the remote planner and coders doing the rest. My own subagent definition for it says it plainly: local and free, but not fast, roughly 22 tokens a second.
It doesn’t replace a hosted model for interactive work, and the reason isn’t the model. It’s the 20,000 tokens of instructions every agent turn starts with, read at 138 a second.
Before LM Studio I ran models through Ollama: Llama 3, small DeepSeek-R1 distills and LLaVA, back in 2024 and early 2025. These days it’s LM Studio on MLX, with the gateway pointing at it.