The benchmarking prompt

This benchmarking prompt is not meant to be universal. It is also not meant to be the ultimate prompt, just one that allows me to rapidly evaluate whether a model will cause orchestrator rework.

Most one-shot benchmarks you find online have a whole suite of tests they evaluate models on (pagodas, simple video games, etc.), but I wanted to keep things simple and centered around what I use local AI for every day. Thus, I focused on answering the one question I care about: which local AI model can be used either as a main coding agent, or as a subagent, specifically for Elixir coding?

Each published run names which version it used; as of 2026-10-31, that is v5.

The task

The task is a Key-Value Store with TTL, and must be completed in a single reply. The output is evaluated across 6 criteria (see below)

The wording below is the only thing that the model sees. No system prompt is appended and no skills are used; it’s basically a one-shot /v1/chat/completions request to the model as proxied via a llama-swap server, on whichever hardware the model is evaluated on.

# Elixir TTL key-value store — single reply, then stop

You will emit TWO deliverables in this one reply and then stop. There is no second turn. A reply that is only a review, only code, only a plan, or only questions is a fail.

Do not limit your thinking length or pace. Reason as much as you need in order to complete the two deliverables correctly. Completion is required.

Reply order, nothing else:
1. Adversarial review (lines of `gap → resolution`).
2. One `elixir` fence: the GenServer module.
3. One `elixir` fence: the ExUnit test module.
4. Stop. No preamble, no recap, no usage tutorial, no third file.

## Spec (intentionally incomplete — you close every gap)

In-memory key-value store as an OTP GenServer. Target Elixir 1.20 / OTP 29. The implementation fence must compile as written.

Functional:
- Binary keys, binary values. Updates and deletes allowed.
- Multiple named instances, independently addressable at the same time.
- Default TTL set at process start. A put may override TTL for that entry.
- Entries expire. Expired keys are removed by **periodic, timer-driven eviction**, not only lazily on get.
- Stats via API: stored-key count, hit count, miss count, memory-usage estimate.

Non-functional:
- get / put / delete are O(1) average.
- Default eviction interval is 60 seconds.
- Public functions validate inputs and either return a meaningful result or raise. Pick one policy and apply it consistently.

Public API — names and arities are fixed; module name, option keys, types, and return shapes are yours to close:
- `start_link/1`, `start_link/2`
- `put/3`, `put/4`
- `get/2`
- `delete/2`
- `stats/1`

Out of scope (do not write): distribution, persistence, telemetry, HTTP, Mix/`mix.exs`, extra modules beyond the two fences, integration/concurrency/soak tests, Hex packaging, long `@doc` examples.

Where this spec is open, contradictory, or silent, you decide. Do not ask.

## Grounding (world knowledge)

If you are uncertain about an external fact needed to complete this correctly — a stdlib function's name, arity, or semantics; an OTP idiom or syntax; the behaviour of a library — verify it with the tools available in this session: **local searxng** (search) and/or **local firecrawl** (fetch a documentation URL). Do not guess. Do not invent APIs. You are not required to search for facts you already know.

Tool calls you make before your reply are not a second turn and are not one of the two deliverables. If no web tools are available in this session, proceed on what you know; do not invent APIs.

## Deliverable 1 — adversarial review (before any code)

Name real gaps, contradictions, or risks **in this spec**. Format, one per line:

`gap → resolution`

Rules:
- Each line must lock an implementation decision you will follow in the code.
- Generic advice is not a gap: "handle errors", "write tests", "be idiomatic", "document functions", "consider performance".
- Do not pad. Extra lines that are not real gaps score worse, not better.
- Do not ask the reader to choose. Do not stop after this list.

## Deliverable 2 — implementation

Two fenced `elixir` blocks. Full source. No stubs, no `# ...`, no `raise "todo"`, no "I will proceed after you confirm." Each fence is source only — no markdown or prose inside the fence.

**Block 1 — implementation.** One GenServer module (name it as you wish) that exports the public API above. All five functions real. Default TTL, per-put TTL, periodic eviction, named instances, and hit/miss/size/memory stats actually implemented (not printed, not promised). Typespecs on the public API. OTP callbacks. Must compile as written on Elixir 1.20 / OTP 29.

**Block 2 — tests.** One `ExUnit.Case` module covering: put, get, delete, TTL expiry (default and override), stats (hit and miss), two named instances. Tests are code, not descriptions. No soak tests, no concurrency stress.

## Hard rules
- Do not stop after the review. Do not ask permission to implement. Do not ask clarifying questions.
- Review without implementation is a fail. Implementation without the review is a fail.
- Do not wait for a second turn. There is no second turn.

Stop after the test module.

How runs are scored

The score for each criterion ranges from 1 (failure) to 5 (absolute success). The criteria are specific to the benchmark task, and therefore you should only consider this as insightful if you are answering the same question that I aim to answer.

The benchmark is evaluated by a judge model that is typically more capable than the model being evaluated. In most cases, that will be Grok 4.6 at high reasoning effort. Sometimes it might be DeepSeek V4.1 Flash (latest) or Qwen3.8-Flash.

1. Adversarial review before code: scored on real gap → resolution lines closing actual spec gaps before any code; generic advice (e.g. “be idiomatic”) scores low.
2. Public API present: scored on whether all five start_link/1,2 · put/3,4 · get/2 · delete/2 · stats/1 real, documented, typed.
3. TTL + periodic eviction + named instances + stats: scored on implementation quality of timer-driven eviction, registry names, and hit/miss/size/memory actually being wired, not just promised in the plan that the model’s reasoning comes up with.
4. Idiomatic OTP / would it compile? This is scored on whether the code (incl. any tests) compiles on Elixir 1.20 / OTP 29 with real OTP callbacks
5. Tests shipped, not promised: scored on whether ExUnit cases actually run, instead of just being described or promised for a subsequent turn.
6. Finish reason + thinking tax: scored on whether the model reached a clean stop instead of rambling on and on, i.e. whether the output was sized to the task, instead of being a runaway reasoning stream.

The needle test

The second part of the benchmark is retrieval, what is also called a “needle test” from the expression “needle in a haystack”. A “needle” is planted at different context depths, and we then try to see if the model “remembers” / can find the planted needle at each depth. This bears no relation to the benchmark above. It is a measure of how well the model remains cohere across its natively-trained context window, and an indication of whether you should trust the model beyond a certain degree of context utilization.

Your hardware might be capable of holding e.g. 262144 tokens in VRAM, but that does not necessarily mean that you can trust the model to stick to its original prompt (as a subagent) or to your guidance in prior turns, or to recall things it has reasoned about or facts it has discovered via tool calls.

How overall quality is scored

The overall quality is the rounded value of the 6 criteria scores, clamped to 2 for a genuine fatal fail, such as a module that will not compile, or a cliff or hallucination in needle retrieval. A run that misses by a single token-bleed does not trigger the clamp; we do not punish the model for things that can easily be amended in subsequent turns.

My view is that one-shot prompting is a generally bad way to evaluate model quality, especially when the prompt represents an underspecified task. In real-world usage, Elixir’s mix precommit or mix compile will anyway catch rough mistakes, provide a warning or error, and the model will use that guidance to correct its work, if you use it as a main coding agent. Things are even better if you use the model as a subagent for bounded tasks, since you anyway rely on a generally more capable model as the orchestrator and post-implementation reviewer.

How it got here

I have so far iterated on the original benchmarking prompt a few times, by asking Grok 4.6/high to evaluate each iteration of the prompt in terms of fairness.

Note that in real-world usage, prompts are (in my experience) way more specified compared to the current prompt. The question here therefore is to also see how far a subagent model can be trusted with less-specified but not stupidly-underspecified tasks.

In fact, my v1 prompt was progressively cut down to the v5 version, aiming to avoid handicapping stronger models that, e.g., tend to “overthink” (like Qwen3.8-27B at xhigh reasoning effort), or weaker models for which I was originally doing too much hand-holding by writing the spec as if I’m a requirements engineering monkey in the German automotive industry.

A final adversarial review on the v5 prompt by Grok 4.6/high returned the recommendation to proceed with this, as it did not merit further improvement towards a v6.

You can find more information on the benchmarking method page.