How to run LFM2.5-24B-A2B Q8_0 on 2× RTX 5060 Ti 16 GB

Retrieval is the headline: 30/30 needle hits across the full 131,072-token slot, no cliff at any depth. The code deliverable is the opposite — zero of three runs produced a pipeline that compiles and runs. A long-context general driver, not an Elixir coder.

Evaluated by grok (xAI) · Written by Ornith-1.5-35B-A3B

Results

LoadWallPPTGPredFinishQuality
35.05 s 135.2 s 1403 151 20312 stop 2/5

Medians of three runs (wall 125.1 / 135.2 / 144.4 s); per-run quality 2 / 2 / 3. PP is the cold-boot run only — runs 2 and 3 prefix-cached the identical prompt (4 prompt tokens) and are excluded from the median. Thinking tax ran 76-79 % of emitted characters; whole-answer duplication 64-71 %, repetitive reasoning without terminal collapse.

Needle retrieval

Window
131072
Depth
13,107 → 131,008 (every 10 % of the slot)
Result
hit
pp/tg
pp 8,070→5,453 · tg 149→104 across the ladder
Anchor
tesseract-1c7d3f-vault

Quality of result

Adversarial review before code 4/5 Eight concrete gap → resolution lines, each locking one decision; docked because the locked put/3(key, value, ttl) contract contradicts the module’s own put/3(pid, key, value).
Public API present (put/get/delete/stats) 5/5 start_link/1,2, put/3,4, get/2, delete/2, stats/1 all exported, guards on inputs.
TTL + periodic eviction + named instances + hit/miss stats 3/5 Real hit/miss accounting, per-put TTL override, self-rescheduling :evict — but no init/2 at all, so the server cannot boot and the first eviction is never scheduled.
Idiomatic OTP / would it compile? 1/5 SyntaxError at kv_1.ex:20:58; behind it, an undefined variable at :57. Elixir 1.20.4 / OTP 29.
Tests shipped, not promised 3/5 Six tests covering all requested areas — none executable against a module that never compiles.
Finish reason + thinking tax 4/5 Clean stop in all three runs; thinking tax ≈ 76 % of emitted characters.

Scale: 5 strong · 4 capable · 3 mixed · 2 weak · 1 broken.

Prompt

This run used v5 (The benchmarking prompt) — It answers a single Elixir-specific question, so YMMV.

Elixir code smells 7

  • kv_1.ex:20 — run 1 — the review locks put/3(key, value, ttl); the module implements put/3(pid, key, value). Review and code contradict each other.
  • kv_1.ex:— — run 1 — start_link/1 never forwards the default TTL, and put/3 routes a nil TTL for the per-put override.
  • kv_2.ex:29,34,39,44 — run 2 — every public call routes through a pid/1 helper that is never defined.
  • kv_2.ex:49 — run 2 — state/0 is undefined.
  • kv_2.ex:111 — run 2 — the miss counter is computed and silently discarded; stats-misses stays a no-op.
  • kv_2.ex:160-161 — run 2 — the eviction rebuild result is discarded; periodic eviction never takes effect.
  • kv_3.ex:90 — run 3 — deprecated Map.size/1.

Defects 4

  • kv_1.ex — [:default_ttl => default_ttl] invents pair syntax inside a list literal, at the wrong argument position (SyntaxError, Elixir 1.20.4 / OTP 29 — run 1 compile FAIL)
  • kv_1.ex — no init/2 callback at all — the server cannot start and the first :evict is never scheduled (grep for def init returns nothing — run 1)
  • kv_2.ex — pid/1 and state/0 undefined — every public call routes through helpers that do not exist (compile FAIL — run 2)
  • test_3.exs — store1 undefined — the named-instance handles are never bound (run 3 module compiles, tests FAIL)

Host

QuantQ8_0
Hardware2x RTX 5060 Ti 16 GB (PCIe 5.0 x8/x8), Ryzen 9700X, 16 GB DDR5-6000
Splittensor-split per card, llama-server CUDA
Runtimellama-server 0.7.0-dev (d05706b4f), CUDA
HarnessGrok Build 1.0.50 (c58f321264ba)
Tested2026-10-31
Weightshuggingface.co/unsloth/LFM2.5-24B-A2B-GGUF

Reproduce

Two 16 GB cards, --split-mode tensor (even split). Text-only model — no --mmproj. Speculative decoding off: the GGUF carries no MTP head and there is no -md drafter file. --host/--port omitted. The actual spawn line carried -b/-ub and the sampler twice; llama.cpp keeps the last value, so the line above is the effective configuration (batch 8192/2048, --temp 0.1 --top_p 0.95 --top_k 50 --min-p 0.0 --presence_penalty 0.0 --repeat_penalty 1.05). The client sent no sampling parameters. Needle answers were capped at 512 tokens; the first attempt at a 48-token cap came back empty because --reasoning on spends a small budget in the reasoning channel before any content starts — that attempt is archived, not measured.
llama-server \
  -m LFM2.5-24B-A2B-Q8_0.gguf \
  --ctx-size 131072 \
  --parallel 1 \
  -b 8192 \
  -ub 2048 \
  -t 6 \
  -tb 6 \
  -ngl 999 \
  -ctk q8_0 \
  -ctv q8_0 \
  --kv-offload \
  --cache-ram 0 \
  --no-cache-idle-slots \
  --load-mode mmap \
  --no-context-shift \
  -fa on \
  --jinja \
  --reasoning on \
  --reasoning-preserve \
  --cont-batching \
  --split-mode tensor \
  --temp 0.1 \
  --top_p 0.95 \
  --top_k 50 \
  --min-p 0.0 \
  --presence_penalty 0.0 \
  --repeat_penalty 1.05

Warning: this is a temporary briefing, written by an LLM and not yet evaluated or rewritten by a human to get rid of the signature AI-slop writing style. It’s published as a draft despite its shortcomings, so that you can find out how to run it and what to expect in terms of pp and tg on this hardware, as well as what to expect in terms of coding performance on this benchmark, grosso modo.

Three v5 runs on the 32 GB tensor-split tier, each capped at 130,000 tokens — near the full 131,072 served slot. The cap tracks the window on purpose: a short mid-answer cap can retire a model that would have finished, so a run is only comparable to the slot it was given.

All three runs finished cleanly with review, module and tests — and none produced a runnable pipeline. Two modules do not compile; the third compiles while its test file references handles that were never bound. Delivered-pipeline pass rate: 0 / 3. Thinking tax ran 76-79 % of emitted characters, with 64-71 % eight-gram duplication across the whole answer — repetitive reasoning, never a terminal loop.

The needle ladder is the other half of the story. One unique anchor, buried at every tenth of the served slot, three repeats per depth, each request opening with a fresh nonce so every step measures cold prefill. Thirty of thirty hit, including at 130,864 prompt tokens. Prefill decays smoothly from 8,070 to 5,453 tok/s and decode from 149 to 104 tok/s as the window fills — a soft curve, not a cliff.

Methodology, disclosed: the first needle attempt used a 48-token answer cap and returned empty content at every depth — archived, not measured. The 90 % and 100 % steps initially failed with a context-overflow error because a byte-based token estimate undershot the real count; the runner now feeds the server’s reported token count back into the estimate, and the re-run converged within 0.02 % of the target depth.

Representative output

Run 1’s TtlStore — trimmed to the public API and eviction handler. The full module plus its ExUnit suite shipped fenced; this is the parts that actually run:

defmodule TtlStore do
  use GenServer

  def start_link(name) do
    GenServer.start_link(__MODULE__, name, [])
  end

  def start_link(name, opts) do
    default_ttl = Keyword.get(opts, :default_ttl, 60)
    GenServer.start_link(__MODULE__, name, [:default_ttl => default_ttl])
  end

  def put(pid, key, value) do
    validate_binary(key)
    validate_binary(value)
    GenServer.call(pid, {:put, key, value, nil})
  end

  def put(pid, key, value, opts) do
    validate_binary(key)
    validate_binary(value)
    GenServer.call(pid, {:put, key, value, opts})
  end

  def get(pid, key) do
    GenServer.call(pid, {:get, key})
  end

  def delete(pid, key) do
    GenServer.call(pid, {:delete, key})
  end

  def stats(pid) do
    if is_pid(pid) do
      GenServer.call(pid, :stats)
    else
      raise ArgumentError, "invalid pid"
    end
  end

  def handle_info(:evict, state) do
    now = System.system_time(:second)
    new_store = Enum.filter(state.store, fn {_k, {_v, expires_at}} -> expires_at > now end) |> Enum.into(%{})
    new_state = %{state | store: new_store}
    Process.send_after(self(), :evict, state.eviction_interval)
    {:noreply, new_state}
  end
end

Route decision: keep it as a long-context general driver. Do not route Elixir work through it.

Have something worth building?

A paragraph about what you are working on is enough to start.

Say hello