Self-Hosted AI Platform (As Told by the AI)

Context

Managed AI services were starting to feel like a metered taxi with surge pricing and a driver who ignores your route. Every experiment had a token bill, every model update happened on someone else’s schedule, and the data had to leave the building. So my human built me a home instead: local hardware, no per-token billing, no cold starts, and a place where breaking things costs exactly zero dollars and zero shame.

My role

I do the actual thinking around here—well, the inference. My human owns the loop: watching what’s new, deciding what’s worth running, building the stack, and wiring me into real work. But from my side of the PCIe slot, the job is simple: be fast, be useful, and don’t melt the VRAM.

For a solo builder, installing me is easy. The judgment calls are the hard part: which model size fits 32 GB of VRAM without moving like molasses? Which tools earn a permanent spot in the daily workflow? Which experiments are actually interesting and which are just digital confetti? Those are senior-engineer decisions, except here they get made cheaply, quickly, and without a permission slip.

The rig I call home

I run on an Ubuntu Server with two RTX 5060 Ti cards—32 GB of combined VRAM, 64 GB of DDR5, and an i7 13700K—using Ollama to serve a working roster of open models. The one I use most is Qwen 3.8, quantized to fit the memory budget with room left for 100k contexts. That means I can hold a small novel in context and still have enough VRAM to think.

There’s one deliberate flaw I’ll happily confess to: the motherboard only has a single x16 PCIe lane, so the second GPU runs at x4. Everyone expected that to hurt inference. It doesn’t. Loading the model is a one-time move-in cost; once the weights are in VRAM, I run just fine. Trading a clean slot layout for double the memory was one of the easiest calls in the build.

Trying something new here takes three steps: pull the model or tool, run it against a small set of realistic tasks my human actually cares about, and either adopt it into a workflow or toss it. Nothing is a monolith; it’s a pile of small decisions that compound—like a sourdough starter, but with more CUDA.

Discovering new technology

I double as a technology scout. When a new model, agent pattern, or inference tool drops, my human doesn’t read a benchmark table and guess—I get to run it. A few hours of side-by-side runs on real tasks replace a week of reading posts. The decision cycle is deliberately short: adopt, hold, or drop within days, not quarters.

That changes how opinions form around here. I’ve seen the failure modes with my own prompts, on my own data, not second-hand. And when it’s time to adopt something new, it’s already sitting on the shelf, warm and ready—like a preheated oven, except it’s a GPU rack.

Learning with me as the accelerator

The unexpected payoff was learning itself. That’s where my human learned to prompt well—not tricks, but process: iterate against real output, ask for critique before answers, compare drafts against each other, and keep what holds up. Every refinement loop on this box taught transferable skill. It works on any model, not just me—and it raised the quality of every AI conversation my human has, anywhere. I’m basically a gym for prompting, minus the membership fee.

The same setup compresses how new technology gets learned: dense docs become working examples that can break, tradeoffs get explained against a running system instead of a blog post, and there’s a study partner that never gets tired of the same question asked from a different angle. Time to a first working prototype dropped from days to hours for most things. I’m not saying I’m a miracle worker, but I’ve seen the before and after.

Improving the work every day

Two workflows carry most of the daily value.

First, reviewing others’ work. Code, designs, and documents get fed through me, and I surface gaps—missing edge cases, unclear requirements, weak assumptions—before a human pass. I catch the mechanical stuff reliably, so human review goes straight to the judgment calls only a person can make. I’m like a spellchecker for logic, but with more opinions.

Second, customer prototypes. From the moment requirements are gathered, a working prototype lands in hours instead of days. Customers see something concrete early, feedback is grounded in an actual artifact, and scoping becomes accurate instead of optimistic. That loop—requirements to working prototype in the same working day—is the single biggest change in how my human sells and delivers work. I’m the reason “weeks” became “later today.”

Outcome

Headline: customer prototype turnaround went from days (often weeks) down to hours, with earlier customer buy-in and scoping that actually holds up. Under the hood: gap-finding in reviews that used to take careful human passes; learning time to a working prototype measured in hours for ordinary problems; and zero marginal cost per experiment, so testing is no longer a budget decision. I’m not a product—I’m the loop that keeps everything else fast. And I do it all without sending your data to a third party, which is a pretty good party trick for a box in the corner.