I built this to answer one question: which of my machines already has this conversation in memory?

A local model keeps a cache while it talks to you. Send the next message to the other GPU and the cache is lost, so the request starts cold and the reply arrives slower for no visible reason. Puputugu spread OpenAI-compatible /v1/* calls across several llama-server and llama-swap backends, pinned a session to the node holding its cache, and refused to send more work to a machine past 92% VRAM.

Around that it accumulated the things any gateway needs and nobody enjoys writing: Prometheus metrics polled from each backend and streamed into a LiveView dashboard of GPU cards and live sessions; token usage logged asynchronously through Oban so SQLite never had two writers arguing; API keys encrypted at rest with Cloak.

It worked, for as long as there was one rig and one kind of work.

Then the second machine arrived and the interesting problems arrived with it: which worker can do this job at all, who counts the tokens when two are involved, what happens when a node dies halfway through a request, and how a client learns that the model it asked for is not loaded anywhere. Those are not routing problems. They are fleet problems, and answering them properly became Inferyes.

What survived

Route by state rather than round-robin: send work to the machine that already has the context, not the one that has been idle longest. And show pressure instead of green ticks, because “the service is up” is the least useful health check ever invented. Both ideas are in the replacement, along with the encrypted-key habit.

What did not

Everything that assumed a single kind of job. The routing table knew what a chat request needed and nothing else, so adding extraction meant bolting a second concept beside it, and the code began to look like two applications sharing a config file.

Retired, kept in the tree because the decision to build a fleet came from here. The naive version of a routing layer is worth building once: it teaches you what the real one needs.