llama.cpp runs open models in the GGUF format on your own machine, on its CPU or its GPU. Its llama-server serves the model you start it with over an OpenAI-compatible API. It’s free and open source, and your own hardware does the work.
Start the server
Section titled “Start the server”-
Install llama.cpp, such as with
brew install llama.cppon a Mac or Linux,winget install llama.cppon Windows, or from its releases. -
Start
llama-serverwith a model, from Hugging Face with-hf, or from a GGUF file with-m:Terminal window llama-server -hf <user>/<model> --jinja
--jinjauses the model’s own chat template, which tool calls need. Newer builds turn it on by default, and giving it does no harm. Some models need--chat-template-filetoo: see llama.cpp’s function calling page.- Port 9931 is where fxbot looks, and where newer builds listen unless told otherwise. Older ones, and llama.cpp’s container images, use 8080.
-c, or--ctx-size, sets the context. Without it, llama-server takes the model’s own.--aliasgives the model the id fxbot shows. Without it, the id is the model file’s path.--api-keymakes it ask for a key. Without it, it takes none.
llama-server listens on 127.0.0.1 unless --host says otherwise. curl http://localhost:9931/props shows the context it runs with, as n_ctx.
Add it to fxbot
Section titled “Add it to fxbot”- In Settings › Providers, choose Add provider. Under On your machine, llama.cpp says Found here when llama-server is running at its usual address with no key. fxbot asks each server there for its model list when the page opens, and gives it half a second to answer.
- Choose llama.cpp. Its address is filled in, with Found running on this machine. under it when fxbot found it. If llama-server runs on another port, change it, such as to
http://localhost:8080/v1for an older build. - Leave API key, if the server needs one empty, unless you started it with
--api-key. Set Context window, and choose Add. fxbot checks the server first. - Pick the model. Tick Make it the default for new bots to start each new bot on it.
| Field | What to put there |
|---|---|
| Address | llama-server’s, with its /v1. The usual one, http://localhost:9931/v1, is filled in. |
| API key, if the server needs one | One of its --api-key keys, if you gave any. It goes to fxbot’s server once, sealed, and never comes back to the page. |
| Context window | The context llama-server runs with: its n_ctx, or the -c you gave it. fxbot can’t read it from llama-server’s list. It applies to every model on this provider. |
| Name | Optional. It’s llama.cpp when you leave it empty. |
When fxbot runs from a checkout, with npm run dev, it runs on the machine itself, and the usual address works as it is.
In a release install
Section titled “In a release install”A release install, the one setup.sh makes, runs fxbot in a container. There, localhost is the container, not your machine, so fxbot doesn’t find llama-server, and the usual address doesn’t reach it. fxbot takes an HTTPS address, or plain HTTP only on localhost. A server on the machine is plain HTTP from inside the container, which fxbot can’t use yet, and the card says so. The same goes for a server on another machine.
Put llama-server behind HTTPS, and give fxbot that address. With Tailscale, and HTTPS turned on for your tailnet, serve it on a port of its own, such as 8443:
sudo tailscale serve --bg --https=8443 9931Then give fxbot https://<machine>.<tailnet>.ts.net:8443/v1 as the Address. fxbot’s own tailscale serve, on port 443, stays as it is (Reach fxbot from other devices).
Its models
Section titled “Its models”fxbot lists the model from llama-server’s own list, /v1/models, by its id: the model file’s path, or its --alias. Newer builds say whether it reads images. The list gives no context window or tool use that fxbot reads, so set Context window yourself.
When it fails
Section titled “When it fails”- “The key check couldn’t reach localhost:9931.” llama-server isn’t running there. Older builds listen on 8080: put its port in the address. In a release install, that’s the container’s own localhost: see In a release install.
- “llama.cpp rejected this key (HTTP 401): it’s wrong, or it was deleted.” llama-server was started with
--api-key, and the key you gave isn’t one of them, or you gave none. - “baseUrl must use https, or http on localhost.” The address is plain HTTP to somewhere other than
localhostor127.0.0.1. Give an HTTPS address. - The model writes tool calls as text, or calls none. Start llama-server with
--jinja. If it still doesn’t, the model may need--chat-template-file, as llama.cpp’s function calling page says.