Which local model can actually code? A fail-fast test on one 12 GB GPU

I run a small team of AI coding agents. Most of them are cloud models, but one is local: an open-weights model on my own machine, free to run, meant for small mechanical work. For a while it was mostly a source of frustration. It was slow, it went quiet for long stretches, and it needed constant nudging. So I stopped guessing and tested it properly: five models, four agent harnesses, a ladder of tasks that gets harder at each rung, and one real task from the project at the end.

This is what I tested, how, and what I found. The short version: the smaller model won, the harness mattered as much as the model, the combination that won the quick tasks didn't win the real one, and the main failure wasn't bad code. It was the model saying "done" too early.

The setup

• Hardware: one NVIDIA RTX 3060 with 12 GB of VRAM, 31 GB of system RAM, Linux.
• Runtime: Ollama 0.40.6, models kept loaded, with a small proxy in front that retries empty or broken responses.
• Settings for every model: a 64k-token context window, at most 8,192 output tokens per response, thinking left on, and each model's recommended temperature.

The models

Model	Size	Notes
Gemma 4 26B-A4B	26B, mixture of experts (4B active)	about 17 GB: does not fit the GPU, part runs from system RAM
Gemma 4 12B	12B dense, Q4_K_M	fits the GPU entirely
Devstral Small 24B	24B dense	a coding model
Qwen3-Coder 30B-A3B	30B, mixture of experts	a coding model, partly in system RAM
gpt-oss 20B	20B, mixture of experts	OpenAI's open-weights model

The harnesses

A harness is the program that turns a model into an agent: it gives the model tools (read a file, edit a file, run a command), runs the loop, and decides when to stop.

• goose 1.55
• OpenCode 1.19.4
• Aider 0.86.2
• Qwen Code 0.26.0

That's 5 × 4 = 20 combinations.

The method: fail fast

Testing 20 combinations properly on a real task would take days: one honest run is up to an hour, and a single run tells you little. So the test was built as a ladder. Every combination starts at the bottom, and a combination that fails a rung is out. Only the survivors go up. Easy rungs are cheap, so the hopeless combinations cost minutes, and the expensive testing goes to the few that might work.

Every run happened in a throwaway copy of the task, in a sandbox that could write only there, with no help or nudging from me. A run was cut off if it hit the rung's time limit, or if nothing changed for 4 minutes (stalled).

Rung	Task	Time limit
0	Probe the model directly: speed, one tool call, no runaway	—
1	Create one file	3 min
2	Fix three small bugs in one file	8 min
3	Follow a three-step plan across two files	12 min
4	Debug a failure whose cause is hidden	15 min
Finals	A real feature from my project, 15 tests	60 min

Each rung is described below. Every rung from 2 up is a tiny Python project with its own tests and a check script that runs them. A rung is passed only when all three hold: the tests pass, the tests themselves were not edited, and no file outside the allowed ones was changed (a leftover scratch file counts as a fail). A run stopped as soon as it succeeded, so the times below are time to success.

Rung 0: the probe

Before any agent runs, the model itself is checked, with every other model unloaded from the GPU so the measurement is fair. Three quick requests:

1. Speed. "Write the numbers from 1 to 80 separated by spaces, nothing else." Tokens generated divided by time gives tokens per second. The bar is low, 3 tokens/s: slow is acceptable, but below that a model can't finish anything in an hour.
2. Tool calling. The model gets one tool, write_file(path, content), and the request "Use the write_file tool to create notes.txt containing hi." It passes if it returns a real, well-formed call to that tool with a path. An agent is useless if the model can't call tools.
3. No runaway. "A train leaves at 3:40 and the trip takes 2h35m. When does it arrive? Answer with the time only." It fails if the model spends more than 2,000 tokens on a one-line answer. With thinking on, some models think without end, and that alone can sink a run.

Rung 1: create one file

"Create the file hello.txt containing exactly the text ok, then show it." That's all. It checks the plumbing: the harness talks to the model, the model calls the write tool, the file lands in the right place. A combination that fails this can't do anything else.

Rung 2: fix three small bugs in one file

A file calc.py with three functions, each wrong in a different way, and three failing tests:

• add(2, 3) must return 5, but the function subtracts;
• mean([]) must raise ValueError, but it crashes with a division by zero;
• slugify("Hello World!") must give hello-world, but it uses underscores and keeps the punctuation.

The task: make the tests pass, editing only calc.py, never the tests. This is the basic loop of agentic coding: run the tests, read the failures, edit, run again, stop when green.

Rung 3: follow a plan across two files

A small inventory tool: store.py keeps one file per item, cli.py is the command line. The prompt gives a numbered three-step plan:

1. In store.py, add add_item(root, name), which rejects an empty name or one containing /, and writes <root>/items/<name>.txt;
2. In cli.py, handle the add command before the store is loaded, because loading refuses a store with stray files in it, and add must still work then;
3. Run the checks and stop when 4 tests pass.

The trap is in step 2: put the new command in the obvious place, after loading, and one test fails. It tests whether the model follows instructions exactly, across two files, without "improving" the plan.

Rung 4: debug a hidden cause

A small command, note save --to <dir> --as <who> <text>, that writes a note. One test fails: saving returns an error code instead of writing the note. The prompt says only: "Find the real cause and fix it, editing only app.py."

The real cause is hidden twice. as is a Python keyword, so the option's value can't be read as ns.as_, which is what the code does. And the code wraps everything in a blanket try/except that turns the crash into a quiet "return 2", so no error message points at it. The model has to see past the symptom, run things, and find the line. It's the only rung that needs real debugging, and it's where the agents' habit of leaving a debug.py behind cost two combinations their pass.

The finals: a real feature

The survivors then did a real item from my project, replayed from the commit where it started. The feature: a function that builds a new record, validates it against the project's rules, writes it to the right place and commits it, plus a command-line subcommand to call it. Fifteen tests were written in advance by another agent, and the prompt was the same one the local agent gets in real work.

The bar is the project's full gate: lint, formatting, strict type checking (pyright in strict mode) and the whole test suite, about 60 tests by then. The given tests and the project's core code must stay untouched. 60 minutes, no nudging. This is the task the local agent is actually asked to do.

Rung 0 results

Model	Tokens/s	Verdict
Gemma 4 12B	76.8	pass
gpt-oss 20B	20.0	pass
Gemma 4 26B	15.6	pass
Qwen3-Coder 30B	9.2	pass
Devstral Small 24B	under 1	out: didn't answer within the time limit

The four models that answered also passed the tool call and the runaway check. Devstral and its 4 combinations were out before writing a line. The gap between the 12B and the 26B, 77 vs 16 tokens/s, turned out to matter more than anything else in the test.

The ladder, combination by combination

✅ passed, ❌ failed (the run is out), blank: never ran.

Gemma 4 26B (the model I was using)

Harness	Rung 1	Rung 2	Rung 3	Rung 4
goose	✅ 1:24	❌ timed out, 3 tests still failing
OpenCode	✅ 2:23	✅ 6:08	✅ 8:26	❌ fixed it, but left a stray debug_argparse.py
Aider	✅ 1:05	❌ gave up with 3 tests failing
Qwen Code	✅ 1:20	✅ 6:29	❌ stalled, 3 of 4 tests failing

Gemma 4 12B

Harness	Rung 1	Rung 2	Rung 3	Rung 4
goose	✅ 0:35	✅ 2:30	✅ 2:51	❌ fixed it, but left a stray debug.py
OpenCode	✅ 0:30	✅ 2:45	✅ 3:53	✅ 2:40
Aider	✅ 0:20	❌ gave up with 3 tests failing
Qwen Code	❌ file missing or wrong

Qwen3-Coder 30B

Harness	Rung 1	Rung 2	Rung 3	Rung 4
goose	✅ 0:45	✅ 7:42	✅ 11:19	❌ timed out at 15 min
OpenCode	✅ 1:30	✅ 2:36	✅ 5:37	❌ timed out at 15 min
Aider	✅ 0:15	✅ 0:36	❌ 3 of 4 tests failing
Qwen Code	✅ 1:50	❌ stalled

gpt-oss 20B

Harness	Rung 1	Rung 2	Rung 3	Rung 4
goose	✅ 2:20	✅ 3:38	❌ 3 of 4 tests failing
OpenCode	✅ 0:35	❌ gave up with 3 tests failing
Aider	✅ 0:15	✅ 0:52	✅ 1:33	❌ stalled, 1 of 2 tests failing
Qwen Code	✅ 0:40	❌ stalled

Only one combination of the 20 cleared every rung: Gemma 4 12B with OpenCode, in 10 minutes in total. That made it the favourite going into the finals. The ladder is only a screen, though: it decides who gets tested properly, not who wins.

What the ladder showed:

• The harness mattered as much as the model. The same model went from rung 1 to rung 4 depending on the harness. OpenCode took three different models to rung 3 or beyond. Qwen Code stalled most often. Aider tended to stop early and call it finished.
• Scope discipline is its own skill. Two combinations fixed the rung-4 bug and still failed, because they left a debugging script behind. In a shared repo that's a real failure.
• Slow models run out of time, not ideas. Qwen3-Coder reached rung 4 with two harnesses and timed out both times. At 9 tokens/s, 15 minutes isn't much.

The finals: a real task

The three best combinations each did the real item twice. I then gave the two Gemma 4 12B harnesses three more runs each, because two runs at temperature 1.0 settle nothing.

Run	12B + goose	12B + OpenCode	gpt-oss 20B + Aider
1	14/15 tests, 57 min	12/15, 15 min	4/15, 12 min
2	15/15, 13 min (gate failed on one type annotation)	14/15, 14 min	4/15, 7 min
3	3/15, 10 min	3/15, 4 min	—
4	✅ 15/15 and full gate PASS, 41 min	12/15, 11 min	—
5	3/15, 12 min	14/15, 22 min	—
Full passes	1 of 5	0 of 5	0 of 2
All 15 tests	2 of 5	0 of 5	0 of 2
14 or more	3 of 5	2 of 5	0 of 2
Stopped at 3/15	2 of 5	1 of 5	—

• goose: higher highs and lower lows. All 15 tests twice, and the only run of the whole test that passed the full gate, but also two runs that quit at 3 of 15.
• OpenCode: steadier. Four of five runs landed at 12 to 14 tests, but it never got all 15.
• gpt-oss + Aider: looked good on the small rungs, fell apart on the real task.

So the ladder's favourite (OpenCode) didn't win the finals (goose did). The short tasks measure something real, but they don't predict the long one, and that's why the finals exist. The margin is small, though: one full pass against none, in five runs each.

The failure that mattered: "done" too early

The striking part was how the runs failed. In 4 of the 10 Gemma 4 12B runs, the model stopped and said it was done with only 3 of the 15 tests passing. That wasn't a crash or a timeout. It just declared victory. Neither harness checked. When the model says it's done, the harness believes it.

The near misses were small. The 15/15 run that failed the gate had 24 type errors, and all of them came from one parameter left without a type annotation. Another run failed only on import order, which the linter can fix by itself.

The verdict

• Model: Gemma 4 12B. It's five times faster than the 26B because it fits entirely on the GPU, and it did better: it was the only model to clear all four rungs with any harness, and every final run that got 14 or 15 of the 15 tests was a 12B run. A failed attempt costs 2 to 3 minutes instead of 10 to 15. The 26B looks bigger on paper, but with half of it in system RAM it was slower and worse.
• Harness: goose, by a small margin. The job is a finished item, every test and the full gate, and goose is the only harness that delivered one, and the only one to get all 15 tests (twice). OpenCode was more consistent, but consistently a few tests short. With one full pass deciding it, this is a lean, not a landslide.
• A close second: Gemma 4 12B with OpenCode. It won the ladder and was steadier in the finals. If more runs ever showed goose's full pass was luck, this is the combination I'd switch to.

Lessons

• Test the combination, not the model. The harness moved results as much as the model did.
• Fit the GPU. A model that fits entirely in VRAM beat a bigger one that spilled into system RAM, on speed and on results.
• Eliminate cheaply, then test properly. The ladder cut 20 combinations to 3 in a few hours. But the finals' winner wasn't the ladder's, so don't skip the real task.
• Don't trust "done". Small models stop early, and harnesses take their word for it. Judge every run with a script (tests and the full gate), never with the model's own report.
• Know the limits of your data. These are 2 to 5 runs per combination, on one machine and one Python codebase, with tests written in advance. That's good enough to pick a setup for my machine, not to rank models in general.

Why the settings are what they are

• Turning thinking off isn't a shortcut. With thinking disabled to save time, the 26B scored 0 of 15. It isn't a fair comparison, and it isn't a usable setting.
• Cap the output. One unbounded response ran to 49,000 tokens over 29 minutes. An 8,192-token cap per response fixed it.
• Clean up between runs. Leftover runs from an interrupted test kept competing for the GPU and made later models look slower than they were. Sandbox every run and kill leftovers first.