AI agents on a home server: one environment for Windows, MacBook and phone

My hardware

I work on Windows, a MacBook and a Linux server. The server is an ordinary computer that doubles as my NASNAS A computer with disks that other devices at home save files to and read files from over the network.NAS A computer with disks that other devices at home save files to and read files from over the network., a network disk available on my home network. It runs all the time because it draws very little power.

What I get from it

From every computer I connect to the server and use the same agents: CodexCodex An AI assistant from OpenAI that works on files on your computer: it reads and changes them and runs programs.Codex An AI assistant from OpenAI that works on files on your computer: it reads and changes them and runs programs., Claude CodeClaude Code An AI assistant from Anthropic for programming that reads project files and runs commands.Claude Code An AI assistant from Anthropic for programming that reads project files and runs commands. or CopilotCopilot An AI assistant from GitHub for writing code, built into editors such as VS Code.Copilot An AI assistant from GitHub for writing code, built into editors such as VS Code. in VS CodeVS Code A free code editor from Microsoft.VS Code A free code editor from Microsoft.. I used VS Code more earlier, when I opened plain code more often than an environment for agents.

The projects live on the server, so they take no disk space on the MacBook or the Windows machine and put no load on either. I never end up with something on one computer and not on the other. I could also keep the project files on GitHubGitHub A website where projects are stored together with their history of changes.GitHub A website where projects are stored together with their history of changes., but one permanent environment is more convenient.

What I like most is that the server sits at home, and from my phone, even away from home, I give commands to an agent running on a computer on my own network.

Docker on Linux

Docker runs a program in an isolated environment called a container. It works great on Linux, in my opinion worse on Windows, and not very well on macOS either. The reason is simple: a container uses the Linux the computer already runs. On Windows and macOS, Docker has to start a virtual machinevirtual machine A program that imitates a separate computer in a window, running another system, here Linux.virtual machine A program that imitates a separate computer in a window, running another system, here Linux. with Linux in the background, so there is an extra layer that eats memory and slows down things like file access. In containers I usually run PythonPython A popular programming language, e.g. for scripts and data analysis.Python A popular programming language, e.g. for scripts and data analysis. with some packages or NodeNode A runtime for programs written in JavaScript, e.g. websites.Node A runtime for programs written in JavaScript, e.g. websites. with a website.

The downside is that container images take a lot of space. Every layer of an image eats disk.

A graphics card and local models

The server has a graphics card that can handle small models. I run models on it that turn speech into text and text into sound (that may be a topic for a separate post), and I host a few LLMsLLM A large language model, an AI program that understands and writes text, e.g. ChatGPT or Claude.LLM A large language model, an AI program that understands and writes text, e.g. ChatGPT or Claude.. Some of my apps use them, so I process data privately, at home.

For writing a program or a demanding analysis I prefer a paid model. I use the local ones for simple, highly repetitive things, because I pay nothing per call. There is no APIAPI A way for one program to request something from another, here paid access to an AI model from your own app.API A way for one program to request something from another, here paid access to an AI model from your own app. bill and no limit that runs out. The cost is electricity, wear on the hardware, and buying that hardware in the first place.

A windowed app instead of the terminal

I prefer graphical apps to an agent in the terminalterminal A window where you type commands as text instead of clicking.terminal A window where you type commands as text instead of clicking.. I take a screenshot, mark what I don't like on a page or in an app, and comment on what to fix. Voice mode is handy too.

The ChatGPT and Claude desktop apps connect over SSH to my home server without trouble. With a VPSVPS A server rented in the cloud that you connect to over the internet and run your programs on.VPS A server rented in the cloud that you connect to over the internet and run your programs on. rented in the cloud it should work the same way. Then my computer uses no resources, the agent keeps working even after I switch it off, and I can look at the result (a program, a page, an analysis) later from my phone.

The biggest downside

By default, over SSH I don't get computer usecomputer use A mode where an AI agent looks at the computer screen and clicks and types by itself.computer use A mode where an AI agent looks at the computer screen and clicks and types by itself. to the extent I would on Windows, a Mac or a desktop Linux. The agent runs shell commands, e.g. curlcurl A command that downloads files and pages from the internet straight from the terminal.curl A command that downloads files and pages from the internet straight from the terminal. to download something from the web, or Docker containers. That works without windows and without a preview.

PlaywrightPlaywright A tool that opens a page in a browser controlled by a program, e.g. to take a screenshot of it.Playwright A tool that opens a page in a browser controlled by a program, e.g. to take a screenshot of it. helps a bit: the agent opens my page, renders it and shows me how it looks. There is also a way around the missing screen. You can run a desktop without a monitor on the server, e.g. a virtual display with XvfbXvfb A program that creates a virtual screen in memory so graphical apps can run on a server with no monitor.Xvfb A program that creates a virtual screen in memory so graphical apps can run on a server with no monitor. or GNOMEGNOME The look of the desktop on Linux, meaning windows, menus and icons.GNOME The look of the desktop on Linux, meaning windows, menus and icons. reachable over VNC, and the agent controls it by looking at screenshots. It needs no graphics forwarding over SSH. I haven't tested this myself yet.