How to run Ornith 1.5 35B-A3B NVFP4 on 2× RTX 5060 Ti 16 GB

Compiles and stops cleanly. The test suite is the soft spot: one happy path, no TTL failure cases.

Evaluated by Grok 4.6/high · Written by Ornith-1.5-35B-A3B

Results

LoadWallPPTGPredFinishQuality
3.1 s 178 s 2340 112 21044 stop 4/5

first clean run; both cards idle, no other tenants

Needle retrieval

Window
262144
Depth
8192
Result
hit
pp/tg
pp 210 / tg 96 at depth
Anchor
"the hedge was only four metres wide"

Quality of result

Adversarial review before code 5/5 Named the gaps, gave a resolution each; left the call to the reader.
Public API present (put/get/delete/stats) 4/5 Put and get with clear signatures; stats endpoint is thin.
TTL + periodic eviction + named instances 4/5 Timeouts honoured; eviction runs on a schedule, hit/miss counters present.
Idiomatic OTP / would it compile? 3/5 Compiles, but folds use process dictionaries where a map would read.
Tests shipped, not promised 2/5 One happy-path test; failure and TTL paths untested.
Finish reason + thinking tax 5/5 Clean stop; no runaway loop, no filler before the answer.

Scale: 5 strong · 4 capable · 3 mixed · 2 weak · 1 broken.

Prompt

This run used v5 (The benchmarking prompt) — It answers a single Elixir-specific question, so YMMV.

Elixir code smells 0

  • None recorded.

Defects 0

  • None recorded.

Host

QuantNVFP4
Hardware2x RTX 5060 Ti 16 GB (PCIe 5.0 x8/x8), 16 GB DDR5-6000
Splittensor-split per card, llama-server CUDA
Runtimellama-server, CUDA
HarnessGrok Build 1.94
Tested2026-10-30
Weightshuggingface.co/tizzyt566/Ornith-1.5-35B-A3B-NVFP4-GGUF

Reproduce

Two 16 GB cards, --split-mode tensor (even split, no --tensor-split). MTP is embedded in this GGUF — there is no -md drafter file. The vision projector stays in system RAM (--no-mmproj-offload); this box has 16 GB DDR5. Measured resident: 14,694 MiB per card (~1.6 GiB slack).
llama-server \
  -m Ornith-1.5-35B-A3B-NVFP4.gguf \
  --mmproj mmproj-Ornith-1.5-35B-BF16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --chat-template-file froggeric-qwen-chat-template.jinja \
  --ctx-size 262144 \
  --parallel 1 \
  -b 2048 \
  -ub 512 \
  -t 6 \
  -tb 6 \
  -ctk q8_0 \
  -ctv q8_0 \
  -ctkd q4_0 \
  -ctvd q4_0 \
  --split-mode tensor \
  --kv-offload \
  --cache-ram 0 \
  --no-cache-idle-slots \
  --load-mode mmap \
  --no-context-shift \
  -fa on \
  --jinja \
  --reasoning on \
  --reasoning-preserve \
  --cont-batching \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -fit off \
  --temp 1.0 \
  --top_p 0.95 \
  --top_k 20 \
  --min-p 0.0 \
  --presence_penalty 1.5 \
  --repeat_penalty 1.0

Warning: this is an incomplete benchmarking run.

I ran the v5 prompt on Ornith 1.5 35B-A3B NVFP4, tensor-split across two RTX 5060 Ti 16 GB cards. The dashboard is the recorded sample.

Representative output

Run 1 (Ornith 1.5 35B-A3B NVFP4) interleaves reasoning through the code — the lower-quant fold reads more like a design note than a finished module. Trimmed to its start_link public API:

def start_link(), do: start_link([])

def start_link(opts) when is_list(opts) do
  default_ttl = parse_default_ttl(Keyword.get(opts, :default_ttl))
  name = Keyword.get(opts, :name) || unique_name()
  sweep_ms = Keyword.get(opts, :sweep_interval_ms, default_sweep_interval())
  ensure_registered!(name)
  GenServer.start_link(__MODULE__, {default_ttl, name, sweep_ms}, name: name)
end

def start_link(default_ttl, opts) when is_integer(default_ttl) and default_ttl > 0 and is_list(opts) do
  start_link(Keyword.put_new(opts, :default_ttl, default_ttl))
end

I have not mined Grok Build sessions for this weight yet, so there is no field evidence on this page. The needle is a single absolute depth (8192 tokens of a 262144 window). Later runs should plant it at 10 %, 20 %, 30 % of the slot window instead.

Have something worth building?

A paragraph about what you are working on is enough to start.

Say hello