Dude, Just Use llama-swap

By: on Oct 18, 2026
Two fluffy brown alpacas in a green field under a grey sky, one close to the camera and one behind it on the right

I have a problem with defending tools I've already configured.

A friend told me about llama-swap. I defended Ollama anyway. I had a NixOS module for it, I had blog posts about it, I had opinions. So dumb. I should have listened.

My homelab AI server now runs every model through llama-swap, and Ollama is gone from it. Here's the short version, then the config.

What llama-swap Is

llama-swap is one small Go program that sits in front of your model servers. You write a YAML file that says, for each model, the exact command that starts it. A request names a model. llama-swap starts that command, waits for it to be healthy, forwards the request, and unloads it later. Ask for a different model and it swaps.

That's the whole idea. One OpenAI-compatible endpoint (it speaks Anthropic's Messages API too), and behind it, processes you wrote down yourself. Usually that's llama.cpp's llama-server, but it can be anything that serves HTTP. I also run whisper.cpp for speech-to-text behind it, and a Python classifier server.

I wrote about Ollama twice: a NixOS Ollama flake in January 2025, and having a coding agent test prompts against a local model in April. Both were true when I wrote them. Ollama is still the fastest way to get a first model answering on a laptop. The agent-as-test-harness idea works the same against llama-swap; swap ollama run for a curl to /v1/chat/completions.

Here's why I moved.

Why I Left Ollama

My repo has the receipts. None of these were dramatic. They were quiet, and that was the problem.

  • The OpenAI endpoint ignored my context size. My bots talk to /v1. One asked for a 262,144-token window; the runner loaded 32,768. Ollama's own OpenAI compatibility docs say the OpenAI API has no way to set context size, and tell you to build a new model from a Modelfile instead.
  • So context became a global setting, and it spilled. The workaround is a server-wide default. At 262,144, a 27B model fit 50 of its 66 layers on the GPU and put the rest in system RAM. No error. It just got slower.
  • The KV cache type is global too. The Ollama FAQ says so directly: one quantization type for every model on the server.
  • Thinking models replied blank. think: false was ignored on /v1, and Gemma put its whole answer in the reasoning field with empty content (ollama#15288; thinking plus tools is ollama#10976). My bots turned reasoning off fleet-wide to get replies at all.
  • Packaging was a project. nixpkgs shipped Ollama 0.24.0 while upstream was at 0.32.8, so I carried my own overlay. Ollama's build now fetches its own pinned llama.cpp at build time, which the Nix sandbox blocks, and without the right backend flag you silently get a CPU-only binary.

Underneath all of it: Ollama's runner has drifted from upstream llama.cpp, so a fix that lands in llama.cpp doesn't automatically land in Ollama. Zetaphor's Friends Don't Let Friends Use Ollama makes that case at length, and my research notes from August cite it. Some of its specific complaints may be fixed by now. Mine were recorded against the versions I ran.

What Got Fixed

Every model's settings are written down

Each llama-swap entry is a full llama-server command. Mine pin the exact Hugging Face repo and quant, the context size, the sampler from the model's own card, and an 8-bit KV cache. Context is per model now: Gemma 4 26B at its native 262,144, a Qwen 27B at 163,840. No global compromise.

Every entry also sets --gpu-layers 99. If a model doesn't fit, llama.cpp refuses and logs it. No more silent spill into RAM.

Speech-to-text joined the same endpoint

whisper.cpp used to sit on the 24 GB card permanently, holding 2.1 GiB beside whatever model was loaded, and every context window was sized around it. Now it's a llama-swap entry. A voice note loads it through llama-swap's /upstream/<model>/ route, and it unloads after five idle minutes. The whole card is the LLM budget.

Old names kept working

Every entry carries its old Ollama tag as an alias, like gemma4:12b. Clients only needed a new URL; the model name they asked for still worked.

The Config

A minimal config.yaml. One chat model, one speech model, captures off:

# Don't keep request/response bodies (see "What Bit Me")
captureBuffer: 0
# Unload any model after 15 idle minutes
globalTTL: 900

models:
  qwen3.5-9b:
    cmd: |
      llama-server --host 127.0.0.1 --port ${PORT}
      --hf-repo unsloth/Qwen3.5-9B-GGUF:Q4_K_M
      --ctx-size 65536 --gpu-layers 99 --flash-attn on
      --cache-type-k q8_0 --cache-type-v q8_0 --jinja
    aliases: ["qwen3.5:9b"]

  whisper:
    cmd: |
      whisper-server --host 127.0.0.1 --port ${PORT}
      --model /models/ggml-large-v3-turbo.bin --convert
    ttl: 300

groups:
  sidecar:
    swap: false
    exclusive: false   # loading whisper doesn't evict the chat model
    members: ["whisper"]

${PORT} is filled in by llama-swap. --hf-repo downloads the GGUF on first request. --convert needs ffmpeg on the PATH. Newer releases document groups under a routing block; the top-level groups key is still accepted for backwards compatibility.

On NixOS there's a services.llama-swap module, and the settings are the same YAML written as Nix:

services.llama-swap = {
  enable = true;
  port = 11435;
  openFirewall = false;   # see "What Bit Me"
  settings = {
    captureBuffer = 0;
    globalTTL = 900;
    models."qwen3.5-9b" = {
      cmd = ''
        ${llamaServer} --host 127.0.0.1 --port ''${PORT}
        --hf-repo unsloth/Qwen3.5-9B-GGUF:Q4_K_M
        --ctx-size 65536 --gpu-layers 99 --jinja
      '';
      aliases = [ "qwen3.5:9b" ];
    };
  };
};

Here llamaServer is a CUDA build: lib.getExe' (pkgs.llama-cpp.override { cudaSupport = true; }) "llama-server". The default package was CPU-only on my pin.

No NixOS? Just use Docker. The README has unified CUDA and Vulkan images:

docker run -it --rm --runtime nvidia -p 9292:8080 \
  -v /path/to/models:/models \
  ghcr.io/mostlygeek/llama-swap:unified-cuda13

Or brew tap mostlygeek/llama-swap && brew install llama-swap, or a binary from the releases page for Linux, macOS, Windows or FreeBSD.

What Bit Me

"Dude, just" doesn't mean free. Five things cost me real time.

The captures endpoint served my prompts to the network. llama-swap keeps recent request and response bodies for its Activity page and serves them at /api/captures/<id> with no authentication. From another machine on my tailnet, with no credentials, one capture came back as 227,795 bytes. Worse, openFirewall = true had opened the port on the home LAN too. That's every bot conversation, readable by any device on the Wi-Fi. The fix was captureBuffer: 0 and closing the firewall. The current example config documents a 10 MB default and says apiKeys is default-allow. Set captureBuffer: 0 or put a key in front before you expose the port.

A TTL unload deadlocked the whole thing. A request arrived while a model was unloading on its timer. After that, no model ever started again. /health said OK the entire time, and everything hung until the service restarted hours later. That's upstream llama-swap#946, fixed in v242. My NixOS stable pin shipped v224, so I run 249 from unstable now. Check your version.

concurrencyLimit: 1 answers with a 429, instantly. I set it thinking it would queue. It rejects. Ollama queued. My bots told users "the model provider is rate-limiting requests" for a server I own. Leave it unset.

The speech model evicted another model. In the default group, loading whisper for a voice note threw out the classifier my bots were about to call, and the five-minute unload meant it was often cold anyway. The sidecar group above is the fix. The full story is in Give Your Agent a System One.

You own the GGUF, including its mistakes. A community Gemma file I picked had no chat template. llama-server fell back to the wrong prompt format, the model's own stop token never matched, and one reply ran 32,725 tokens. Swapping to the official Google GGUF fixed it. Ollama's library packages the template inside its own model layers. With llama-swap, picking the file is your job.

One more honest note. Ollama is gone from the AI server. On my laptop and my other box, the removal is committed to config but not deployed yet.

Just Use It

If you have one GPU and more than one model, write the commands down and put llama-swap in front of them. You'll know what's on your disk, what each model runs with, and what happens when two things want the card.

And when a friend tells you about a tool, maybe try it before you defend the old one.

The line in this post that should worry you is the captures endpoint: recent bot conversations, readable without a password by any device on the home Wi-Fi, while every health check said OK. Most systems I get called into have a corner like that. Finding it before launch is cheap; finding it after costs whatever leaked or stalled in the meantime. That's the job of a production readiness report from JustinTime Systems: go open the folders and endpoints nobody has opened in months. Here's how I build the map.

Header photo by Josiah Farrow on Unsplash.

Content on this blog was created using human and AI-assisted workflows described in my standards and workflow posts. Original ideas and editorial decisions by Justin Quaintance.