The benchmark method

In my experience since I got into agentic coding with local models in late May 2026, a benchmark measures the prompt as much as the model. You can find out about specifics regarding the benchmarking prompt on the v5 prompt page. This right here is more of a high-level view.

The task

Build a Key-Value Store with TTL as an OTP GenServer. The test is a single user message without a system prompt or skills loaded. Models that I can currently run across my llama-swap servers range up to 74B (MoE). All models get the exact same benchmark treatment.

The reply

The model answers once, in this order:

1. an adversarial review of the spec,
2. one elixir Markdown codefence with the module,
3. one elixir fence with the tests…

…then, stop. No second turn is afforded to patch the output. This is left up to the orchestrator model (if you are running the model being benchmarked as a subagent) or up to other prior guidance (skills, rules, earlier turns) that instruct the model to run QA actions such as mix precommit and catch and correct its mistakes.

What gets recorded

Load time, wall-clock seconds, prefill / prompt processing (pp), decode / token generation (tg), predicted output length, why the stream stopped, and a quality score q. Speed and quality sit next to each other, so a model that finished fast without thinking is not scored as if it did more work.

The “needle test” results are also recorded. This is a “needle in a haystack” test that evaluates coherence/retrieval. The depth should be a percentage of that slot’s context window (-c), not one absolute token count. A single 8192-token plant on a 262k context window is not a retrieval curve. We do this in 10% increments.

Guardrails on the quality of the prompt

The following five rules were part of the process that led to the current v5 prompt:

• Provide no truisms. Common sense you would never put in an AGENTS.md stays out. No “elixirmaxxing” instructions or other nudges like “think like Jose Valim”.
• Don’t overspecify. Handing a small model the answer by means of providing it with the exact things it should do compresses the range of things you are evaluating, and fictionally shrinks the gap between models. It is true that in real-world situations a model used as a subagent will receive a much better specified prompt than what we use now. If you use it as the main agent, sometimes you might not have gone through a V-model approach, and therefore we are also testing whether the model can deduce what it should do, i.e. plan its way out of the ambiguity.
• Don’t underspecify. On the other hand (the flipside of “don’t overspecify”), leaving too much open means that the task becomes practically unbounded. Thus, you’d grade “world knowledge” and “creative range” (I know… to anthropomorphize a bunch of matrices is human) that a smaller model may not have. For example, expecting planning capabilities from MiniCPM5-1B is probably misplaced optimism.
• Don’t nudge to one solution. Leave it up to the model to pick things like the module shape, any Elixir modules it can use, or the algorithm. We evaluate how well the model did on this criterion, regardless of whether it’s a dense 1B or a 74B Mixture-of-Experts model.
• Record tool use without punishing silence. A model that already knows Map.delete/2 and compiles may skip a web search with e.g. a local SearXNG / web fetch with e.g. a local Firecrawl instance. In fact, if a model chooses to use either or both, that’s a positive sign. A smaller model that has a more limited “world knowledge” searching for online answers is a positive thing. A larger model that might know enough about Elixir but might reason that it is uncertain about whether Elixir 1.20 (more recent than its training cut-off date) and proceeds to search online is similarly positive.

Note that the last part is actually not entirely applicable on the v5 benchmark or the method, because a single turn will not be able to call any tools. But it might reason about needing to call tools within its single turn.

Caveats

• This is still a one-shot benchmark. At least it’s focused on answering a single question. Still, perhaps affording the model up to N turns would be a better method.
• The prompt is thin in places I would normally spell out (and that I did spell out in the v1 of the prompt), because spelling everything out for every model at once means being biased towards smaller models, which are exactly the models I’m interested in running as subagents.
• This is not meant to evaluate a model in general, so don’t come flaming me with “hurrr durrr but this model is better than you think” or otherwise. The testing approach asks whether a model ships a compiling product to a fixed brief, and whether it gets Alzheimer’s across the context window that the quant and KV cache quantization settings afford for the hardware.

The prompt itself, and how it evolved from v1 to v5, lives on its own page.