The Pitch Nobody Finishes
Every other post about "running your own coding models" ends right where the hard part starts. They get one model loaded on one box, paste a screenshot of it saying hello, and call it a homelab. Victory lap, roll credits.
The actual goal is messier. I have three machines with NVIDIA cards sitting around: a big one, a mid-range one, and a small one. I want my coding agent, OpenCode, to talk to all three. Pick the big model when I need brains, the small one when I need speed, the middle one for a second opinion. No cloud. No per-token meter running while I think.
Turns out the model part is easy. The plumbing is where you lose an afternoon.
The Shape of It
Three boxes, three jobs:
The big card (high VRAM) runs Qwen 3.6 35B-A3B through vLLM in Docker. Another mixture-of-experts model, 35B total but only 3B active per token, so it's fast for its size. The workhorse.
The mid-range card runs Gemma 4 26B-A4B through llama.cpp. A mixture-of-experts model, so it's 26B total but only 4B active per token. Punches above its size.
The small card (8GB) runs Gemma 4 E4B through llama.cpp. Tiny, fast, good enough for the cheap stuff.
At a glance:
Role GPU Model Context Server
Workhorse RTX 5090 (32GB) Qwen 3.6 35B-A3B 131K vLLM (Docker)
Mid RTX 5080 (16GB) Gemma 4 26B-A4B (MoE) 64K llama.cpp
Fast/cheap RTX 4060 (8GB) Gemma 4 E4B 64K llama.cpp
The boxes aren't in the same room or even the same network. They're stitched together with Tailscale, which gives every machine a stable address on a private network and a hostname that just works from anywhere. That part is genuinely great.
OpenCode points at all three as separate providers. Each one is an OpenAI-compatible endpoint on port 8086. You pick the model in the switcher, the request routes to whichever box owns it.
The config is three provider blocks, each pointing at a Tailscale hostname. The apiKey is required by the OpenAI client but ignored by the servers, so it can be anything:
"provider": {
"local": {
"npm": "@ai-sdk/openai-compatible",
"name": "Local vLLM (5090)",
"options": { "baseURL": "http://localhost:8086/v1", "apiKey": "anything" },
"models": { "Qwen3.6-35B": { "name": "Qwen 3.6 35B-A3B" } }
},
"remote": {
"npm": "@ai-sdk/openai-compatible",
"name": "Remote vLLM (5080)",
"options": { "baseURL": "http://socb25:8086/v1", "apiKey": "anything" },
"models": { "gemma": { "name": "Gemma 4 26B-A4B (5080)" } }
},
"aux4060": {
"npm": "@ai-sdk/openai-compatible",
"name": "Aux (4060)",
"options": { "baseURL": "http://sohome:8086/v1", "apiKey": "anything" },
"models": { "gemma-e4b": { "name": "Gemma 4 E4B (4060)" } }
}
},
"model": "local/Qwen3.6-35B"socb25 and sohome are Tailscale MagicDNS names, not raw IPs. That matters later.
llama.cpp, Not Ollama
I went with llama-server from llama.cpp instead of Ollama on the two Windows boxes. Ollama is friendlier, but it wraps llama.cpp anyway, and it eats a little more VRAM doing it. On an 8GB card you feel every megabyte. llama.cpp gives you the OpenAI server built in, GGUF quants you can swap freely, and tighter control.
One sharp edge worth flagging. winget install ggml.llamacpp pulls the Vulkan build. Vulkan runs on NVIDIA cards, but it's slower than CUDA and it allocates memory worse. I found that out when the mid-range card OOM'd loading a model that should have fit. The fix is to skip winget and grab the CUDA release zip directly off GitHub, plus the matching cudart zip for the runtime DLLs. Unzip both into the same folder. The exe needs those DLLs sitting next to it or it won't start.
Now the trap inside the trap. The CUDA build comes in flavors tagged by CUDA version: 12.x, 13.x, and so on. That version has to be no newer than your GPU driver supports. I grabbed the 13.3 build for the small card and ran it. No error. The model loaded. It said it was listening. And it was running entirely on the CPU.
The only hint was one line at the very top of the log, before all the happy "model loaded" noise: ggml_cuda_init: failed to initialize CUDA: (null). Then it shrugged and fell back to the processor. The card's driver topped out at CUDA 12.9; the 13.3 binary needed newer. Mismatch, silent demotion, four times slower, no complaint.
The fix was a one-line check I should have run first. nvidia-smi prints the max CUDA version the driver supports in the top corner. Match your build to that number, not to the newest one on the releases page. I pulled the CUDA 12 build instead, dropped it in a clean folder so it wouldn't mix with the 13.3 DLLs, and the next launch finally listed the card as a CUDA device instead of just the CPU. Up from 18 tokens a second on the processor to 66 on the card.
The lesson generalizes past llama.cpp: when a tool offers a "GPU" build and quietly works anyway, confirm it's actually on the GPU before you trust the speed. Silent CPU fallback is the worst kind of bug, because nothing is broken. It just isn't doing what you think.
While you're there: the --flash-attn flag changed. It used to be a bare switch. Now it takes a value. -fa on. If you pass it the old way it swallows the next argument and throws a cryptic error about an unknown value. Cost me a minute of squinting.
The Firewall Fight
Here's the part that ate the afternoon.
I get the small box running. Start llama-server, bound to 0.0.0.0:8086, model loaded, server listening. From my main box I curl it over Tailscale. Works first try. 69 tokens a second. Beautiful.
Then I do the exact same thing on the second box. Same command shape, same port, same Tailscale. The server says it's listening. From the box itself, curl localhost:8086 returns the model list. From the box's own Tailscale IP, it works. From my main box across the tailnet, it times out.
Not "connection refused." Timeout. The packets go into the void.
So I do what you do. I check the obvious. Is the server bound right? Yes, 0.0.0.0. Is the host even reachable over Tailscale? Yes, RDP on port 3389 answers fine from across the tailnet. So the machine is there, the route is there, the server is there, and one specific port is being eaten.
That smell is firewall. I add an allow rule for inbound TCP on 8086. Still times out. I make it wider, allow from any address, any profile. Still times out. I go hunting for a Block rule that might be winning, because in Windows Firewall a block always beats an allow. Nothing. No block rule on the port, no block rule on the app.
At this point I stop guessing and run the one test that actually splits the question. Turn the firewall off for ten seconds, curl, turn it back on. It works with the firewall off. So it is the firewall. My allow rules just weren't catching the traffic.
The thing I'd missed: the Tailscale adapter is a virtual network interface, and a generic "allow port 8086" rule wasn't binding to it the way I assumed. The first box worked by luck of a different config. The rule that finally stuck scopes the allow directly to the interface:
New-NetFirewallRule -DisplayName "llama-tailscale" `
-Direction Inbound -Action Allow `
-Protocol TCP -LocalPort 8086 `
-InterfaceAlias Tailscale -Profile Any
-InterfaceAlias Tailscale is the whole fix. Bind the rule to the adapter Tailscale traffic actually arrives on, and stop trying to reason about network profiles. The moment I ran that, the port opened across the tailnet and stayed open with the firewall up.
Lesson I'm keeping: on Windows, when you're serving a port to Tailscale peers and a normal port rule isn't catching it, scope the rule to the interface. Don't burn an hour on profiles like I did.
The Context Trap
Models loaded, firewall solved, all three reachable. I run a smoke test through OpenCode. "Say hello in one word." The big card answers. The other two throw:
Error: request (43138 tokens) exceeds the available context size (16384 tokens)
Forty-three thousand tokens to say hello. That's not my prompt. That's the agent's system prompt. OpenCode's default build agent ships a heavy block of instructions and tool schemas, and it doesn't compress. The big card swallowed it because Qwen was loaded with 131K of context. The smaller boxes were running 16K and 32K, so they choked before my message even mattered.
Worse, OpenCode tried to be helpful. It saw the overflow, ran its auto-compaction to shrink the conversation, retried, overflowed again, compacted again. An infinite loop of the agent summarizing a conversation that was just the word "hello," generating fake progress reports about media attachments that never existed. Genuinely surreal to watch.
The fix is blunt: give the small models more context. I restarted the two smaller boxes with -c 65536. Sixty-four thousand tokens clears the 43K floor with room to work. On the small card that's free, because E4B is a small model and the KV cache fits in 8GB easily. On the mid-range card it's tight, and I'm leaning on Windows shared-memory spillover to hold it. If long sessions start dragging, the move is to drop the vision projector (--no-mmproj, since a coding agent doesn't need image input) and claw back a couple gigabytes.
After the context bump, all three answered through OpenCode. "Hello." "Yo." Qwen said "greet math. hug science," which is its own kind of charming.
What I'd Tell Past Me
If you're wiring a coding agent to local models across machines, the models are the easy 80%. Budget your time for the other 20%:
Get the CUDA build, not Vulkan, and match its version to your driver. winget hands you Vulkan. The newest CUDA build hands you a silent CPU fallback if your driver is older than it wants. Check
nvidia-smi, match the number, then confirm the card shows up in the device list before you trust the speed.Scope firewall rules to the Tailscale interface. Generic port rules quietly fail on virtual adapters.
Match context size to your agent, not your vibe. A heavy agent has a heavy system prompt. Undersize the context and you don't get an error so much as a breakdown.
Use the hostnames. Tailscale MagicDNS gives every machine a stable name. Put those in your config instead of raw IPs so a reboot doesn't break everything.
The invocations I actually run, in one place. The two llama.cpp boxes:
# mid-range card (5080) — Gemma 4 26B-A4Bllama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:Q4_K_M \
--alias gemma -ngl 999 -c 65536 -fa on --jinja \
--host 0.0.0.0 --port 8086
# small card (4060) — same line, swap the model and drop the vision projectorllama-server -hf unsloth/gemma-4-E4B-it-GGUF:Q4_K_M \
--alias gemma-e4b -ngl 999 -c 65536 -fa on --jinja --no-mmproj \
--host 0.0.0.0 --port 8086
The big card runs Qwen 3.6 35B-A3B under vLLM in Docker (GPTQ-Int4 weights, fp8 KV cache, the qwen3_coder tool-call parser, 131K context), managed through a compose file rather than a one-liner. The llama.cpp boxes are where every gotcha in this post lived, so that's where the detail is.
The payoff is real. Three cards, one agent, no cloud bill, pick your model by the job in front of you. The big one thinks, the small one sprints, and the middle one is there when I want a second read on something. That's the setup I actually wanted, and it works.
It just didn't work the way the thinkpieces promised.
— Sean