This is a small AI platform on machines I already own: open models on a Mac desktop and on my home server’s graphics card, one gateway in front of them, and a router that decides where a task should go. Each model was chosen for its job by a written rule over fixed tests, before anything was installed as a service. Nextcloud’s Assistant and my code editor use it today, from the home network only.
The problem
I wanted models that run at home, for work that shouldn’t leave the house and for tools I use every day, and later a group of agents that hand work to each other. The obvious approach is to pick models by reputation, one per machine, and point every tool at every machine. That fails three ways. A model’s reputation says nothing about how fast it runs on a given machine at a long context. Tools wired to machines all break when a model moves. And agents that decide by themselves whether to act need someone checking them.
What I built
- A measurement harness: one pinned llama.cpp build and the same model files, checked by checksum, run on each machine, recording memory, graphics and speed.
- Fixed tests per job: coding problems, tool calls, pulling fields out of documents, questions about page images, search and routing. A written rule turns the results into a table of which model does which job on which machine.
- Model services. On the Mac, the larger models start at boot with nobody logged in, restart after a crash and refuse requests without a key. Two small models stay loaded on the home server’s graphics card.
- One gateway (LiteLLM) on the home server, reachable only from the home network. Each tool gets its own key and limits, and asks for a job by name.
- A router. Given a task, it answers three typed questions: which of the planned agents should take it, whether a person must approve it, and how much work it is.
How it works
A tool sends an OpenAI-style request to the gateway, naming a job such as coder, general or embed. Only the gateway knows which machine serves that job, so moving a model is one change in one place. Code and general questions go to gpt-oss-20b on the Mac, and questions about documents and screenshots to Qwen3.5-9B beside it. Routing and embeddings go to two small models on the home server’s RTX 3060. Image generation and video conversion stay on the home server, where the files are; transcription and upscaling run on the Mac, which measured faster at both.
The router asks Clef, Cloudflare’s decision model on Workers AI, through Cloudflare’s AI Gateway. If Cloudflare fails, a small local model answers instead, and the task never leaves the house. A task can also be pinned to the home network from the start.
A smoke test that reruns the measurement tests through the gateway matched every figure measured directly, and the gateway added 0.061 s to the median time to the first token. Uptime Kuma checks each model server and the gateway, and Prometheus collects their metrics.
Decisions and trade-offs
- Models chosen by a rule, not by preference. Each job’s rule was written before the results were read: for code, the most coding problems passed, with near-ties going to the faster model. It picked gpt-oss-20b over larger models that scored the same, because it was faster in less memory. The cost was measuring first, before anything was usable.
- One gateway with a key per tool. Talking to each machine directly would save a moving part, but every model move would break every tool, and nothing would limit one tool’s load. With the gateway, the editor’s code index has its own key, so its bursts can’t starve the chat.
- A hosted decision model, with a local fallback that asks a person. Running Clef at home isn’t practical: it’s a large model with a custom decision head. So routing goes to Cloudflare, with the text of real requests kept out of its logs, and a small local model takes over when Cloudflare can’t answer. That model caught only 9 of 24 tasks that needed a person’s approval, so any decision it makes will ask a person.
- Two small models kept on a shared graphics card. The card already serves Plex and image generation. In a test with both models loaded and a video transcode running, image generation was about a quarter slower. I accepted that, so routing and search never wait for a model to load.
What’s next
Agents: first a trial of where they run, on Cloudflare or on the home server, then a concierge that routes and hands work on, and agents for code, documents, images and video. Then scheduled evaluations, a dashboard, and new machines measured before they get a job.
Built with
llama.cpp, gpt-oss-20b, Qwen3.5, nomic-embed-text, whisper.cpp, Real-ESRGAN, ComfyUI, LiteLLM, Cloudflare Workers AI, Clef, Cloudflare AI Gateway, Docker Compose, launchd, Python, Uptime Kuma, Prometheus.