Money for Nothing, Tokens for Free: Sovereign AI on a Budget Redux

I wrote recently about the box I built to run my own AI models at home. That provides some background on the what and why. Since then I changed the two biggest software pieces in the stack, and this followup covers the how. The tl;dr is that I tore out llama.cpp and moved everything back to vLLM, and I swapped the Qwen 3.6 models for a recently released model called Ornith 1.0.

The free tokens part is that it’s the same machine from the original article, tweaked a bit, with just a few weeks of software landscape changing to make the juice worth the squeeze. Such is the state of the shifting AI landscape today.

Why I dumped llama.cpp

The original setup ran four separate llama.cpp servers, one pinned to each card carrying two “slots” of Qwen3.6-27B at 128K context. It worked pretty well, but every card was its own island. A request that landed on card 0 could not borrow a byte of memory or a cycle of compute from card 1, even when card 1 sat idle. In effect, I had eight smallish context windows instead of a unified pool, and while I had concurrency for agents and opencode I had no batching across the fleet.

vLLM solves this with something called tensor parallelism. Instead of four copies of the model, you run one copy split across all four cards, and every request goes through a single engine with continuous batching. One large memory pool, one queue, with four cards working together on each token instead of racing each other for the same job. The reason I skipped it the first time was not subtle, but suffice it to say it would not build or run properly on these cards when I started out.

Getting vLLM onto gfx908

MI100 is CDNA1, which is old enough that most of the ROCm world has moved on. Mainline vLLM assumes newer cards and newer math libraries, so a stock build either fails to compile or starts up and then falls over the first time it hits a matrix shape the kernels do not cover.

What actually got me running was a community image built specifically for gfx908, with the AITER kernels patched in. I had investigated this previously but there were blockers that caused me to abandon it at the time. Recent updates made a revisit viable and it has made a night and day difference. With the new image in place, the tuning came down to two facts about these cards that are worth writing down if you ever try this:

The cards cannot do bf16. Try to load a model in bfloat16 and vLLM errors out on gfx908. You need to run everything in plain fp16 (--dtype half) instead. The quality is fine, but you just have to know to ask for it.

I don’t currently have a fast link between the cards. On a proper datacenter setup the GPUs would talk to each other over a dedicated infinity fabric. Because I don’t have the $800 bridge spanning the cards, the peer-to-peer path that would let them copy directly over PCIe is unavailable so I have to turn it off with (NCCL_P2P_DISABLE=1). That means when the four cards combine their partial results on every token, the data goes out to system RAM and back. It sounds slow, and it’s certainly not free, but with a four-way split on a single host it keeps up fine.

What the box actually runs now

So now I have a single vLLM instance, with tensor-parallel across all four cards, serving Ornith 1.0 35B at a 256K context window (--max-model-len 262144), in full FP16 precision, with prefix caching on and the Qwen tool and reasoning parsers wired up so tool calls and chain-of-thought come back cleanly. A small gateway sits in front and hands out the OpenAI-style endpoint everything else talks to. For a single request cold startup, it decodes around 60 tokens a second which is amazing at FP16 precision. And for concurrency, the batching immediately earns its keep allowing 128 requests in flight that clears north of 1,300 tokens a second in aggregate (tokens for free.. they were there before but being wasted).

The model weights eat about 70GB across the four cards. What is left, roughly 40GB, becomes the KV cache, and vLLM reports that as a pool of just over a million tokens. How many things that serves at once depends entirely on how much context each one is using. At the full 256K window it is about four in parallel. At 8K it could be over a hundred, but between my always on agents and two opencode sessions four is the sweet spot.

For what this thing does in a day, roughly 1,500 requests, it never comes close to filling that pool. In the time I’ve been recording metrics on it, the scheduler has never once had to pause and swap a request out for lack of room. The queue wait sits in the microseconds. I would say this configuration finally unlocks the true potential of this hardware, and it’s important to note again that it’s only recently become possible because the software ecosystem has continued to improve.

Why Ornith instead of Qwen

The engine swap was largely mechanical, but the model swap needs a bit more justification.

Ornith 1.0 is a family of open models from DeepReinforce, released June 2026 (approximately a month ago as of this writing), built for agentic coding and shipped under an MIT license. The interesting idea behind it is that during training the model learns to build its own scaffolding, the plan-and-retry loop around a task, at the same time it learns to write the code, rather than being wrapped in a fixed harness a human wrote.

The practical reason I switched is simple. The 35B MoE I run now is the same size and the same mixture-of-experts shape as the Qwen model it replaced, so it was close to a drop-in, but it is meaningfully smarter at the work on the same hardware. On Terminal-Bench 2.1 it scores 64.2 against Qwen 3.6 35B’s 41.4, and on SWE-Bench Verified it is 75.6 against 70. My favorite line from their numbers is that Ornith 1.0 35B beats Qwen’s own 397B model on Terminal-Bench, 64.4 to 53.5. The model card has a lot more details along these lines if you’re into that sort of thing.

The key unlock though is in how to serve it which is where the community shined again. The config I landed on is more or less the recipe DeepReinforce publishes for running it under vLLM, but you don’t really get the performance until you load Froggerics jinja template. And now we know, and knowing is half the battle!

What’s next

These changes have really made a difference on the hardware and make the overall investment well worth it. And for now its serving my needs really well, but I’m already thinking about where I could take it next when I have more money for nothing burning a hole in my pocket.

Ornith also ships a 397B model. That one scores 77.5 on Terminal-Bench and 82.4 on SWE-Bench Verified, which puts it right on top of Claude Opus 4.7. Running it at home means eight cards instead of four, in a chassis built for it, and quantizing the model down to 4-bit to squeeze it into roughly 196GB so it fits across eight 32GB cards. Priced out, before any haggling, it is about $8700 on top of $4k worth of cards already invested. While that is more than the entire original build cost, the chassis alone is ~$4800 before adding memory/disk. I’m currently eyeing an Asus ESC8000A-E11 which has dual EPYC CPUs and can even fit 8 bigger NVIDIA GPUs, so the chassis gives future room to grow if better cards hit a reasonable price point. Here’s what 8 MI100 cards would look like in it:

The model target keeps moving because the models keep getting better, with new ones releasing every few months. But an additional $8700 buys me a much bigger model I own outright that I never have to worry about being deprecated or repriced from under me, and one that is roughly one generation behind whatever I could rent by the token today. Tomorrow that same hardware could run something that is beating Opus 4.8 today, so it’s hard to look at it and say “but another $8700 buys a lot of Claude Max!”. It does, at a different cost. Whether that trade is worth it is the whole question, and it is the same question the first article was really about: what is it worth to own the thing instead of renting it.