Skip to content

The computer

Copy all docs

The computer plugin’s tools, skills, desktop, settings, setup, and image, and what a computer reaches. To add it, see Give bots a computer.

Bots the plugin is on for get seven tools. A relative path starts at /root.

Tool Input What it does
computer_shell command, and timeout in seconds: 120 unless given, 600 at most Runs one bash command in a new shell in /root, and answers exit <code> and its output. A command still running at its timeout is stopped, with its process group, and the output says so. It reaches the desktop, so agent-browser and agent-desktop work from it.
computer_terminal text, typed as is; keys, named, such as Enter, Tab, Escape, Up, or C-c; wait, in seconds: 1 unless given, 30 at most Sends them to the terminal, the tmux session main, waits, and answers its last 200 lines. For a program that asks and waits: a REPL, an installer’s prompts, ssh, top
computer_desktop action, and what it needs, below One action on the screen, then a screenshot
computer_read_file path, offset (the first line, from 1), limit (lines, up to 2,000) Text, 2,000 lines or 60 KB at a time, saying where the next part starts. An image (PNG, JPEG, GIF, or WebP, up to 3.75 MiB) comes back as an image, which the bot sees.
computer_write_file path, and text (up to 1 MiB) or library (a library:<id>); or old_text and new_text Writes the text, or the library item’s bytes, making the folders it needs. With old_text, which must be in the file exactly once, changes that one place to new_text. Either way, a file that was there keeps its mode.
computer_to_library path, and name Saves an image (up to 3.75 MiB) or a text file (up to 1 MiB) to the bot’s library, tagged for the chat, and answers its library:<id>
computer_ask_owner reason, in one line Hands the desktop to the owner for a step only they can do, as Take over does, and shows them the reason

computer_desktop’s actions, in the screenshot’s own pixels:

action Needs
screenshot, cursor_position Nothing
left_click, right_click, middle_click, double_click, triple_click coordinate, [x, y], or the cursor where it is; text for keys held during the click, such as shift or ctrl+shift
mouse_move coordinate
left_click_drag start_coordinate and coordinate
scroll scroll_direction, up, down, left, or right; scroll_amount, 3 unless given; coordinate
type text
key text, a key or combination, such as Return or ctrl+l
wait duration in seconds, 1 unless given, 30 at most
zoom region, [x1, y1, x2, y2]: that part of the screen at its own size

Each answer says what it did, the screen’s size, the active window’s title, and the cursor, with the screenshot, a PNG, or a JPEG when a PNG would be over 3.75 MiB. screenshot: false leaves out the image. A model that can’t see images gets the text, and libfx’s note that the image wasn’t sent. In a reply, a computer_desktop call shows its screenshot; fxbot keeps each bot’s last 50 in memory, so a reply from before fxbot restarted shows its text only.

While the owner has a computer, after Take over or computer_ask_owner, computer_desktop and computer_terminal answer that the owner has it. The shell and file tools go on. When the owner hands back after the bot asked, the bot’s chat gets a message from the computer, and the bot’s turn starts again.

  • Long output. Output over 60 KiB comes back as its first 4 KiB and last 56 KiB, with a line saying how much was cut, and where the whole of it is, in /tmp/fxbot/. A tool’s result can be at most 64 KiB.
  • Servers. A command started with setsid cmd > log 2>&1 & keeps running after its call, until the computer stops.
  • Calls at once. A reply’s shell commands run one at a time, and its reads and writes run together. Different actors’ calls, such as a bot’s subagents’, run at once.
  • Skills. When a bot reads a skill with files besides SKILL.md, its files are copied to /skills/<name>/ on the bot’s computer before its next call, and the skill’s text says so.

Each chat of a bot the computer is on for has a few lines in its instructions: which part of the computer to use for what, cheapest first, and to ask the owner before it sends, buys, signs in, or deletes. Other bots’ chats have none.

The plugin has three skills:

Skill What it covers
computer Which tool for which job, and how to act on a screen and check what happened. $computer in a message, or the screen button beside the message box, sends it with the message.
browser Web pages with agent-browser 0.41.1, the version on the computer: how to open, read, fill, click, wait, and take screenshots, what to leave as it is on this computer, and what to do when a step fails. agent-browser skills get core has the rest.
desktop Desktop apps with agent-desktop 1.1.0, by their accessibility trees: the commands, refs, when it’s safe to try a step again, and when to use computer_desktop instead. agent-desktop skills get desktop has the rest.

Each computer is named for its bot, as its hostname, such as bob-computer, and runs a desktop: XFCE on an X display, :1, with a panel, the Thunar file manager, Chromium, and a terminal window on the tmux session main. Its VNC server listens on a unix socket in the computer, /tmp/fxbot/vnc.sock, and no computer listens on a TCP port.

  • The Computer screen, /computer/computers/<key>, shows the desktop through noVNC and the terminal through xterm.js, over WebSockets at /api/plugins/computer/desktop/<key> and /api/plugins/computer/terminal/<key>. A bot’s key is its id, and /computer/bots/<id>, from before 0.3.0, opens the same screen. The desktop is view-only until Take over; the terminal takes typing always. For a stopped computer, the screen has Start it.
  • Cursors. Over the desktop, each bot, or each of a bot’s labelled sessions, that pointed at something has an arrow in its colour, with its name, what it last did, and how long ago. The newest is solid, and the others faded. A bot that last used the browser or an app’s accessibility tree, which move no pointer, shows as a label at the top left. They come from computer_desktop’s clicks, moves, drags, and scrolls, kept in memory, and cleared when the computer stops or is reset. Since 0.3.0.
  • The browser is Chromium, run as root without its sandbox, with its profile in /root/.config/chromium, so sign-ins last. The agent-browser command starts it on the desktop when it isn’t running, brings its window to the front, and drives it over DevTools on 127.0.0.1:9222, inside the computer.
  • A bot’s sessions at once, such as its subagents, since 0.4.0. A session a plugin labelled is an actor of its own on the bot’s computer, named “ ·
  • Accessibility is on for every app, so agent-desktop reads their trees.
  • Fonts are DejaVu and Noto, which cover most scripts, but not Chinese, Japanese, Korean, or color emoji.

Since computer 0.3.0, a plugin that needs the computer can name computers of its own, such as a room’s. Each is named by a key, a word, a dash, and more, such as room-1a2b3c4d, which a bot’s id never is. The plugin provides, with fx.provide:

type ComputerProvider = {
/** The computer for a chat, or null for the bot's own. `name` makes its hostname: `<name>-computer`. */
computerFor(call: { bot: string; session: string }): Promise<{ key: string; name: string } | null>;
/** Every computer it keeps, with its name; null while it can't tell. */
computers(): Promise<{ key: string; name: string }[] | null>;
};
  • Which computer a call uses. Before each call, the computer asks each such plugin, in turn, for the chat the call is in. The first that names one wins; otherwise the bot’s own is used.
  • Its container and home are fxbot-computer-<key>, labelled fxbot.by=<plugin>.
  • Keeping it. A plugin’s computer goes when that plugin is removed, or when it runs and its computers() doesn’t list the key. One added but not running, or whose computers() answers null, keeps its own.
  • Shared. The bots it names it for share it, taking turns. A computer_desktop result there says where the others last pointed: “Others on this computer: Ann clicked (412, 300) 2 min ago.” Take over pauses every bot’s desktop and terminal tools on it, and Hand back sends nothing: the owner tells the bot that asked to go on.
  • Routes, under /api/plugins/computer/computers/<key>/, for the page: state, cursors, start, stop, reset, and takeover. A key no running plugin lists, and no bot’s id, gets 404.

On its tab in Settings:

Setting Default What it is
engine /run/fxbot-computer/engine.sock, then /var/lib/fxbot-computer/run/engine.sock The engine’s socket
memoryMiB 2048 Memory for each computer, in MiB
cpus 2 CPUs for each computer
idleMinutes 15 Minutes without a call before a computer stops. One someone watches doesn’t stop.
screen 1280x800 The desktop’s size, width x height in pixels

Each computer also has at most 512 processes, and 512 MiB of shared memory for browsers. A computer made with other sizes, or another screen, is made again with the new ones when it next starts; its home stays.

The computer’s engine is set up by host/setup.sh, in the plugin’s files, run as root. When the plugin loads and finds no engine, it has the script run, once each time fxbot starts:

fxbot runs Who runs the script
From a release install The machine’s fxbot command, with fxbot computer on, which the plugin asks for through fxbot’s requests. It runs the script from the release’s own files, not the plugin’s folder, then starts fxbot again.
From a checkout, as root The plugin, with --user root
From a checkout, as another user Nobody: Settings › Computer shows the command, to run as root

Settings › Computer shows the setup while it works, and why it failed, with Try again. FXBOT_COMPUTER_NAME names another user than fxbot-computer, for checks beside a real install.

The script’s options, for running it by hand:

Option What it does
none A release install in /opt/fxbot, and compose.computer.yaml there, which gives fxbot’s container the socket’s folder and the user’s group. Before 0.3.2 it wrote compose.override.yaml: it takes its own lines out of that, and leaves yours.
--user <login> A checkout: <login> joins the user’s group, to reach the socket
--engine podman or --engine docker This engine. Otherwise the one the user already runs rootless; then podman, when the distribution has 4.9 or newer; then the machine’s own Docker, when its packages run it rootless
--name <user> Another user than fxbot-computer
--install <folder> Another release install than /opt/fxbot
--remove Takes it all away: the user, its computers and files, the firewall, and compose.computer.yaml. The packages stay.

What it sets up:

Thing Where
Packages Through apt, dnf, zypper, or pacman, each listed in /etc/fxbot/packages: nftables, and curl and util-linux when they’re missing; on apt, uidmap and dbus-user-session; with podman, podman and passt, and slirp4netns for podman 4; with docker, docker-ce-rootless-extras beside Docker’s docker-ce, rootlesskit beside Debian’s docker.io, moby-engine-rootless-extras beside Fedora’s moby-engine, or docker-rootless-extras beside openSUSE’s docker, and slirp4netns; where SELinux runs, policycoreutils-python-utils
The user fxbot-computer, home /var/lib/fxbot-computer, 65,536 subordinate ids, lingering on
Limits /etc/systemd/system/user@<uid>.service.d/fxbot-computer.conf delegates cpu, memory, and pids to it
The engine Its socket at /var/lib/fxbot-computer/run/engine.sock. Podman starts when called and stops 5 seconds after; Docker’s rootless daemon runs all the time, and holds about 180 MB.
The firewall /etc/fxbot/fxbot-computer.nft, loaded at boot by fxbot-computer-firewall.service
SELinux Where it runs, podman’s storage in the user’s home takes /var/lib/containers’ labels (semanage fcontext -e), and fxbot’s container runs with its labels off, to reach the socket

It stops before it changes anything when the machine runs cgroups v1, without which the limits would do nothing, or keeps its users from making user namespaces. Where podman is older than 4.9 and Docker can’t run rootless, as with Ubuntu 22.04’s docker.io, it stops with the fix: Fix an install that stopped.

The firewall matches the connections of the user fxbot-computer, which are every connection its computers make.

Reached Refused
The internet This machine, at any of its addresses
DNS, through the machine’s resolvers Its tailnet, 100.64.0.0/10 and fc00::/7
Private networks: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16, and fe80::/10
Loopback, multicast, and fxbot’s containers

A refused connection fails at once. To let a computer reach one address on your network, add a rule before the others in the computers chain of /etc/fxbot/fxbot-computer.nft, such as ip daddr 192.168.1.20 tcp dport 8080 accept, then run sudo systemctl restart fxbot-computer-firewall.

Every computer runs from one image, localhost/fxbot-computer:<install>-<hash>, about 1.2 GB, made from Debian 13 slim with:

  • ca-certificates curl git python3 procps ripgrep jq unzip xz-utils
  • the desktop: TigerVNC’s Xvnc, XFCE, Thunar, xfce4-terminal, tmux, xdotool, scrot, and the accessibility bus
  • chromium, and fonts: DejaVu and Noto
  • agent-desktop 1.1.0 and agent-browser 0.41.1, each checked against its SHA-256

A running computer holds about 120 MB at idle, and about 330 MB with a page open in Chromium.

Two fxbots on one machine, such as a checkout beside a release, share the engine. Each labels the computers, homes, and images it makes with an id of its own, kept in the plugin’s table, and lists, stops, deletes, and prunes only its own. A computer another fxbot made for a bot is left alone, and a call for it says so.

A plugin that needs the computer adds setup steps, which run as root with the network, in name order, after the computer’s own. The computer’s files go in last, so a step can’t replace them. The image is made again in the background when such a plugin is added, updated, or removed; a running computer moves to it the next time it starts. An image no computer uses is removed.