The computer plugin’s tools, skills, desktop, settings, setup, and image, and what a computer reaches. To add it, see Give bots a computer.
Bots the plugin is on for get seven tools. A relative path starts at /root.
| Tool | Input | What it does |
|---|---|---|
computer_shell |
command, and timeout in seconds: 120 unless given, 600 at most |
Runs one bash command in a new shell in /root, and answers exit <code> and its output. A command still running at its timeout is stopped, with its process group, and the output says so. It reaches the desktop, so agent-browser and agent-desktop work from it. |
computer_terminal |
text, typed as is; keys, named, such as Enter, Tab, Escape, Up, or C-c; wait, in seconds: 1 unless given, 30 at most |
Sends them to the terminal, the tmux session main, waits, and answers its last 200 lines. For a program that asks and waits: a REPL, an installer’s prompts, ssh, top |
computer_desktop |
action, and what it needs, below |
One action on the screen, then a screenshot |
computer_read_file |
path, offset (the first line, from 1), limit (lines, up to 2,000) |
Text, 2,000 lines or 60 KB at a time, saying where the next part starts. An image (PNG, JPEG, GIF, or WebP, up to 3.75 MiB) comes back as an image, which the bot sees. |
computer_write_file |
path, and text (up to 1 MiB) or library (a library:<id>); or old_text and new_text |
Writes the text, or the library item’s bytes, making the folders it needs. With old_text, which must be in the file exactly once, changes that one place to new_text. Either way, a file that was there keeps its mode. |
computer_to_library |
path, and name |
Saves an image (up to 3.75 MiB) or a text file (up to 1 MiB) to the bot’s library, tagged for the chat, and answers its library:<id> |
computer_ask_owner |
reason, in one line |
Hands the desktop to the owner for a step only they can do, as Take over does, and shows them the reason |
computer_desktop’s actions, in the screenshot’s own pixels:
action |
Needs |
|---|---|
screenshot, cursor_position |
Nothing |
left_click, right_click, middle_click, double_click, triple_click |
coordinate, [x, y], or the cursor where it is; text for keys held during the click, such as shift or ctrl+shift |
mouse_move |
coordinate |
left_click_drag |
start_coordinate and coordinate |
scroll |
scroll_direction, up, down, left, or right; scroll_amount, 3 unless given; coordinate |
type |
text |
key |
text, a key or combination, such as Return or ctrl+l |
wait |
duration in seconds, 1 unless given, 30 at most |
zoom |
region, [x1, y1, x2, y2]: that part of the screen at its own size |
Each answer says what it did, the screen’s size, the active window’s title, and the cursor, with the screenshot, a PNG, or a JPEG when a PNG would be over 3.75 MiB. screenshot: false leaves out the image. A model that can’t see images gets the text, and libfx’s note that the image wasn’t sent. In a reply, a computer_desktop call shows its screenshot; fxbot keeps each bot’s last 50 in memory, so a reply from before fxbot restarted shows its text only.
While the owner has a computer, after Take over or computer_ask_owner, computer_desktop and computer_terminal answer that the owner has it. The shell and file tools go on. When the owner hands back after the bot asked, the bot’s chat gets a message from the computer, and the bot’s turn starts again.
- Long output. Output over 60 KiB comes back as its first 4 KiB and last 56 KiB, with a line saying how much was cut, and where the whole of it is, in
/tmp/fxbot/. A tool’s result can be at most 64 KiB. - Servers. A command started with
setsid cmd > log 2>&1 &keeps running after its call, until the computer stops. - Calls at once. A reply’s shell commands run one at a time, and its reads and writes run together. Different actors’ calls, such as a bot’s subagents’, run at once.
- Skills. When a bot reads a skill with files besides
SKILL.md, its files are copied to/skills/<name>/on the bot’s computer before its next call, and the skill’s text says so.
Skills and instructions
Section titled “Skills and instructions”Each chat of a bot the computer is on for has a few lines in its instructions: which part of the computer to use for what, cheapest first, and to ask the owner before it sends, buys, signs in, or deletes. Other bots’ chats have none.
The plugin has three skills:
| Skill | What it covers |
|---|---|
computer |
Which tool for which job, and how to act on a screen and check what happened. $computer in a message, or the screen button beside the message box, sends it with the message. |
browser |
Web pages with agent-browser 0.41.1, the version on the computer: how to open, read, fill, click, wait, and take screenshots, what to leave as it is on this computer, and what to do when a step fails. agent-browser skills get core has the rest. |
desktop |
Desktop apps with agent-desktop 1.1.0, by their accessibility trees: the commands, refs, when it’s safe to try a step again, and when to use computer_desktop instead. agent-desktop skills get desktop has the rest. |
The desktop
Section titled “The desktop”Each computer is named for its bot, as its hostname, such as bob-computer, and runs a desktop: XFCE on an X display, :1, with a panel, the Thunar file manager, Chromium, and a terminal window on the tmux session main. Its VNC server listens on a unix socket in the computer, /tmp/fxbot/vnc.sock, and no computer listens on a TCP port.
- The Computer screen,
/computer/computers/<key>, shows the desktop through noVNC and the terminal through xterm.js, over WebSockets at/api/plugins/computer/desktop/<key>and/api/plugins/computer/terminal/<key>. A bot’s key is its id, and/computer/bots/<id>, from before 0.3.0, opens the same screen. The desktop is view-only until Take over; the terminal takes typing always. For a stopped computer, the screen has Start it. - Cursors. Over the desktop, each bot, or each of a bot’s labelled sessions, that pointed at something has an arrow in its colour, with its name, what it last did, and how long ago. The newest is solid, and the others faded. A bot that last used the browser or an app’s accessibility tree, which move no pointer, shows as a label at the top left. They come from
computer_desktop’s clicks, moves, drags, and scrolls, kept in memory, and cleared when the computer stops or is reset. Since 0.3.0. - The browser is Chromium, run as root without its sandbox, with its profile in
/root/.config/chromium, so sign-ins last. Theagent-browsercommand starts it on the desktop when it isn’t running, brings its window to the front, and drives it over DevTools on127.0.0.1:9222, inside the computer. - A bot’s sessions at once, such as its subagents, since 0.4.0. A session a plugin labelled is an actor of its own on the bot’s computer, named “
· - Accessibility is on for every app, so
agent-desktopreads their trees. - Fonts are DejaVu and Noto, which cover most scripts, but not Chinese, Japanese, Korean, or color emoji.
A computer for another plugin
Section titled “A computer for another plugin”Since computer 0.3.0, a plugin that needs the computer can name computers of its own, such as a room’s. Each is named by a key, a word, a dash, and more, such as room-1a2b3c4d, which a bot’s id never is. The plugin provides, with fx.provide:
type ComputerProvider = { /** The computer for a chat, or null for the bot's own. `name` makes its hostname: `<name>-computer`. */ computerFor(call: { bot: string; session: string }): Promise<{ key: string; name: string } | null>; /** Every computer it keeps, with its name; null while it can't tell. */ computers(): Promise<{ key: string; name: string }[] | null>;};- Which computer a call uses. Before each call, the computer asks each such plugin, in turn, for the chat the call is in. The first that names one wins; otherwise the bot’s own is used.
- Its container and home are
fxbot-computer-<key>, labelledfxbot.by=<plugin>. - Keeping it. A plugin’s computer goes when that plugin is removed, or when it runs and its
computers()doesn’t list the key. One added but not running, or whosecomputers()answers null, keeps its own. - Shared. The bots it names it for share it, taking turns. A
computer_desktopresult there says where the others last pointed: “Others on this computer: Ann clicked (412, 300) 2 min ago.” Take over pauses every bot’s desktop and terminal tools on it, and Hand back sends nothing: the owner tells the bot that asked to go on. - Routes, under
/api/plugins/computer/computers/<key>/, for the page:state,cursors,start,stop,reset, andtakeover. A key no running plugin lists, and no bot’s id, gets 404.
Settings
Section titled “Settings”On its tab in Settings:
| Setting | Default | What it is |
|---|---|---|
engine |
/run/fxbot-computer/engine.sock, then /var/lib/fxbot-computer/run/engine.sock |
The engine’s socket |
memoryMiB |
2048 | Memory for each computer, in MiB |
cpus |
2 | CPUs for each computer |
idleMinutes |
15 | Minutes without a call before a computer stops. One someone watches doesn’t stop. |
screen |
1280x800 |
The desktop’s size, width x height in pixels |
Each computer also has at most 512 processes, and 512 MiB of shared memory for browsers. A computer made with other sizes, or another screen, is made again with the new ones when it next starts; its home stays.
The setup
Section titled “The setup”The computer’s engine is set up by host/setup.sh, in the plugin’s files, run as root. When the plugin loads and finds no engine, it has the script run, once each time fxbot starts:
| fxbot runs | Who runs the script |
|---|---|
| From a release install | The machine’s fxbot command, with fxbot computer on, which the plugin asks for through fxbot’s requests. It runs the script from the release’s own files, not the plugin’s folder, then starts fxbot again. |
| From a checkout, as root | The plugin, with --user root |
| From a checkout, as another user | Nobody: Settings › Computer shows the command, to run as root |
Settings › Computer shows the setup while it works, and why it failed, with Try again. FXBOT_COMPUTER_NAME names another user than fxbot-computer, for checks beside a real install.
The script’s options, for running it by hand:
| Option | What it does |
|---|---|
| none | A release install in /opt/fxbot, and compose.computer.yaml there, which gives fxbot’s container the socket’s folder and the user’s group. Before 0.3.2 it wrote compose.override.yaml: it takes its own lines out of that, and leaves yours. |
--user <login> |
A checkout: <login> joins the user’s group, to reach the socket |
--engine podman or --engine docker |
This engine. Otherwise the one the user already runs rootless; then podman, when the distribution has 4.9 or newer; then the machine’s own Docker, when its packages run it rootless |
--name <user> |
Another user than fxbot-computer |
--install <folder> |
Another release install than /opt/fxbot |
--remove |
Takes it all away: the user, its computers and files, the firewall, and compose.computer.yaml. The packages stay. |
What it sets up:
| Thing | Where |
|---|---|
| Packages | Through apt, dnf, zypper, or pacman, each listed in /etc/fxbot/packages: nftables, and curl and util-linux when they’re missing; on apt, uidmap and dbus-user-session; with podman, podman and passt, and slirp4netns for podman 4; with docker, docker-ce-rootless-extras beside Docker’s docker-ce, rootlesskit beside Debian’s docker.io, moby-engine-rootless-extras beside Fedora’s moby-engine, or docker-rootless-extras beside openSUSE’s docker, and slirp4netns; where SELinux runs, policycoreutils-python-utils |
| The user | fxbot-computer, home /var/lib/fxbot-computer, 65,536 subordinate ids, lingering on |
| Limits | /etc/systemd/system/user@<uid>.service.d/fxbot-computer.conf delegates cpu, memory, and pids to it |
| The engine | Its socket at /var/lib/fxbot-computer/run/engine.sock. Podman starts when called and stops 5 seconds after; Docker’s rootless daemon runs all the time, and holds about 180 MB. |
| The firewall | /etc/fxbot/fxbot-computer.nft, loaded at boot by fxbot-computer-firewall.service |
| SELinux | Where it runs, podman’s storage in the user’s home takes /var/lib/containers’ labels (semanage fcontext -e), and fxbot’s container runs with its labels off, to reach the socket |
It stops before it changes anything when the machine runs cgroups v1, without which the limits would do nothing, or keeps its users from making user namespaces. Where podman is older than 4.9 and Docker can’t run rootless, as with Ubuntu 22.04’s docker.io, it stops with the fix: Fix an install that stopped.
What a computer reaches
Section titled “What a computer reaches”The firewall matches the connections of the user fxbot-computer, which are every connection its computers make.
| Reached | Refused |
|---|---|
| The internet | This machine, at any of its addresses |
| DNS, through the machine’s resolvers | Its tailnet, 100.64.0.0/10 and fc00::/7 |
Private networks: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16, and fe80::/10 |
|
| Loopback, multicast, and fxbot’s containers |
A refused connection fails at once. To let a computer reach one address on your network, add a rule before the others in the computers chain of /etc/fxbot/fxbot-computer.nft, such as ip daddr 192.168.1.20 tcp dport 8080 accept, then run sudo systemctl restart fxbot-computer-firewall.
The image
Section titled “The image”Every computer runs from one image, localhost/fxbot-computer:<install>-<hash>, about 1.2 GB, made from Debian 13 slim with:
ca-certificates curl git python3 procps ripgrep jq unzip xz-utils- the desktop: TigerVNC’s
Xvnc, XFCE, Thunar,xfce4-terminal,tmux,xdotool,scrot, and the accessibility bus chromium, and fonts: DejaVu and Noto- agent-desktop 1.1.0 and agent-browser 0.41.1, each checked against its SHA-256
A running computer holds about 120 MB at idle, and about 330 MB with a page open in Chromium.
Two fxbots on one machine, such as a checkout beside a release, share the engine. Each labels the computers, homes, and images it makes with an id of its own, kept in the plugin’s table, and lists, stops, deletes, and prunes only its own. A computer another fxbot made for a bot is left alone, and a call for it says so.
A plugin that needs the computer adds setup steps, which run as root with the network, in name order, after the computer’s own. The computer’s files go in last, so a step can’t replace them. The image is made again in the background when such a plugin is added, updated, or removed; a running computer moves to it the next time it starts. An image no computer uses is removed.