Why run your own harness when you pay for a subscription

What a harness is and why I set one up

An AI model on its own only answers with text. A harness is a program around it: it gives the model tools (a browser, files, a search engine), memory and tasks that run at a set time, and you talk to it through a chat app. OpenClaw and Hermes are two such open-source programs that you run yourself. They promise an assistant that runs around the clock and does things on its own.

I didn't have a problem for such an assistant to solve. I set them up out of curiosity and because the "24/7 assistant" was getting a lot of attention at the time. I invented the use caseuse case A specific way of using a tool, for example "summarises my mail every morning".use case A specific way of using a tool, for example "summarises my mail every morning". to see how it works, and I suspect plenty of people do exactly the same.

What I tried and on what

I ran everything on models on my own computer: Gemma 4Gemma A family of AI models from Google whose files you can download and run on your own computer.Gemma A family of AI models from Google whose files you can download and run on your own computer. and Qwen 3.6Qwen A family of AI models from Alibaba that you can also run on your own computer.Qwen A family of AI models from Alibaba that you can also run on your own computer., in 9Bmodel parameters (9B, 27B) The number of settings inside an AI model, where B means billions. Usually more means a more capable model, but also a heavier one.model parameters (9B, 27B) The number of settings inside an AI model, where B means billions. Usually more means a more capable model, but also a heavier one. versions. I picked them because they cost nothing beyond electricity and hardware I already had, so I don't pay for tokenstoken A piece of text, such as a word or part of one, that a model splits what it reads and writes into. With a paid API, the price depends on the number of tokens.token A piece of text, such as a word or part of one, that a model splits what it reads and writes into. With a paid API, the price depends on the number of tokens.. I talked to the agentagent A program built on an AI model that carries out tasks by itself, for example searching, clicking and saving.agent A program built on an AI model that carries out tasks by itself, for example searching, clicking and saving. through DiscordDiscord A chat app for text and voice conversations in groups, popular among gamers.Discord A chat app for text and voice conversations in groups, popular among gamers., which I use to reach people anyway, so the choice was obvious.

The first attempt was plot listings. I had the agent search for them and it went badly: it couldn't load pages or search properly. I set up the SearXNGSearXNG A search engine you run yourself. It sends your question to several other search engines and collects their results.SearXNG A search engine you run yourself. It sends your question to several other search engines and collects their results. search engine myself, and ExaExa A search engine for programs and AI agents that returns the clean text of pages.Exa A search engine for programs and AI agents that returns the clean text of pages., built for AI agents, was an option too.

The second was a reminder. The agent was supposed to message me on Discord until I explicitly replied that I had done something. It didn't work, because the program was full of bugsbug A mistake in a program that makes it behave differently from what it should.bug A mistake in a program that makes it behave differently from what it should..

The third was controlling it by voice from my phone through Open WebUIOpen WebUI A web page for talking to AI models running on your own machine, looking much like ChatGPT.Open WebUI A web page for talking to AI models running on your own machine, looking much like ChatGPT.. It practically didn't work.

What kept breaking

Programs like this have wide permissions on the system, so I ran them in rootless Dockerrootless Docker Docker runs a program in an isolated environment. The rootless version has no administrator rights, so the program cannot break the rest of the computer.rootless Docker Docker runs a program in an isolated environment. The rootless version has no administrator rights, so the program cannot break the rest of the computer. to keep a mishap away from the rest of the computer. In that mode they barely started at all. I lost time on permissions, digging through GitHub issuesGitHub issues The list of bug reports and ideas of a project on GitHub, the service where programmers keep their code.GitHub issues The list of bug reports and ideas of a project on GitHub, the service where programmers keep their code. and one workaround after another. On top of that, the frontendfrontend The part of a program you see and click: windows, buttons, forms.frontend The part of a program you see and click: windows, buttons, forms. saved settings wrongly.

The longest hunt was around scriptsscript A short program, a list of commands carried out one after another.script A short program, a list of commands carried out one after another.: the agent wrote them itself to launch tools and validated them itself, and I spent half a day debuggingdebugging Finding and fixing mistakes in a program.debugging Finding and fixing mistakes in a program. why it didn't work. In the end it turned out the model I was using was simply too weak.

Some of the blame is the models', because those from a few months ago were a lot worse than today's. I also suspect these projects are tested less than apps from big companies, but that's only a guess. I didn't try the harnesses with a cloud model, so the conclusion about weak models applies to these small local ones.

What the provider apps have

The one real advantage of these harnesses is proactivity: the agent wakes up on its own (heartbeatheartbeat A regular signal that wakes the agent so it can check whether it has anything to do.heartbeat A regular signal that wakes the agent so it can check whether it has anything to do.) and does tasks at a set time. You can already schedule a task for a given hour in the Codex and Claude apps, and the providers are going further, toward agents that run in the cloudcloud Computers owned by companies in data centres that you use over the internet instead of running the program yourself.cloud Computers owned by companies in data centres that you use over the internet instead of running the program yourself. all the time: Google has Gemini SparkGemini Spark An AI agent from Google that runs in the cloud around the clock and, for example, watches your mail and calendar.Gemini Spark An AI agent from Google that runs in the cloud around the clock and, for example, watches your mail and calendar. (May), OpenAI has Dotsdots An AI agent from OpenAI that lives in ChatGPT, has its own computer in the cloud and works around the clock.dots An AI agent from OpenAI that lives in ChatGPT, has its own computer in the cloud and works around the clock. (September), and on October 21 Anthropic showed Claude DuetClaude Duet An app with an AI agent from Anthropic that suggests what to do on its own, e.g. plans meeting times.Claude Duet An app with an AI agent from Anthropic that suggests what to do on its own, e.g. plans meeting times.. According to reports, all of it is still in betabeta An early version of a product that a company gives to selected users to test.beta An early version of a product that a company gives to selected users to test. or in selected countries, mostly the US, so it isn't available here yet. I'd guess because of data processing regulations.

The search built into Claude and Codex is simply better for me than a hand-wired SearXNG. Since I pay for the model anyway, why assemble the same thing from scratch.

When your own harness makes sense

The first case is hardware that can handle a good model. That means a Mac StudioMac Studio A small desktop computer from Apple that can be bought with a very large memory, which AI models need a lot of.Mac Studio A small desktop computer from Apple that can be bought with a very large memory, which AI models need a lot of. with 256 GB of memory, or a mini computer built on Nvidia chips (DGX SparkDGX Spark A small computer from Nvidia for running AI models at home.DGX Spark A small computer from Nvidia for running AI models at home., not to be confused with Gemini Spark). Models that run on such machines are where OpenClaw or Hermes start to make sense. Gemma and Qwen 9B were too weak on my computer, mainly because they can't cope with tool callingtool calling A model's ability to run a tool by itself, such as a search engine.tool calling A model's ability to run a tool by itself, such as a search engine., and the agent's whole job rests on calling tools by itself. They become usable from around 27B, and by my estimate a machine like that costs at least 20 to 30 thousand zlotys. That is enough for simple scripts, looking up information and simple analysis.

Writing code and more complicated work still need the cloud models from providers. Hostinghosting Keeping a program running on a computer that is always on and available to users.hosting Keeping a program running on a computer that is always on and available to users. a model with 700B+ parameters makes no sense cost-wise, unless it's a big company that will keep such a machine busy 24/7. For me the value of that hardware is, for now, too low to beat a 100-zloty subscription, which is roughly what the cheapest useful plan costs at most providers. Then there's contextcontext The amount of text a model can see at once. Everything sent to it takes up room.context The amount of text a model can see at once. Everything sent to it takes up room.: the harness fills it with a lot of its own promptsprompt A text with an instruction that you send to an AI model.prompt A text with an instruction that you send to an AI model. (memory, instructions, scheduled tasks), so you have to fit it to the model.

The second case is data that can't leave for a provider, legal data for example. Then your own model is what's left. At work I have such a case: the agent gets full permissions so it can read the files given to it, and it has no internet apart from the model's APIAPI An interface through which one program asks another for data or to do something.API An interface through which one program asks another for data or to do something., so that nothing leaks in an uncontrolled way. The Codex and Claude apps are simpler here: fetching a page usually needs no approval, while sending something to a form usually does, though it depends on the permission settings.

Related: a website an agent can use and an assistant with access to your apps.