Skip to content

llama.cpp

Copy all docs

llama.cpp runs open models in the GGUF format on your own machine, on its CPU or its GPU. Its llama-server serves the model you start it with over an OpenAI-compatible API. It’s free and open source, and your own hardware does the work.

  1. Install llama.cpp, such as with brew install llama.cpp on a Mac or Linux, winget install llama.cpp on Windows, or from its releases.

  2. Start llama-server with a model, from Hugging Face with -hf, or from a GGUF file with -m:

    Terminal window
    llama-server -hf <user>/<model> --jinja
  • --jinja uses the model’s own chat template, which tool calls need. Newer builds turn it on by default, and giving it does no harm. Some models need --chat-template-file too: see llama.cpp’s function calling page.
  • Port 9931 is where fxbot looks, and where newer builds listen unless told otherwise. Older ones, and llama.cpp’s container images, use 8080.
  • -c, or --ctx-size, sets the context. Without it, llama-server takes the model’s own.
  • --alias gives the model the id fxbot shows. Without it, the id is the model file’s path.
  • --api-key makes it ask for a key. Without it, it takes none.

llama-server listens on 127.0.0.1 unless --host says otherwise. curl http://localhost:9931/props shows the context it runs with, as n_ctx.

  1. In Settings › Providers, choose Add provider. Under On your machine, llama.cpp says Found here when llama-server is running at its usual address with no key. fxbot asks each server there for its model list when the page opens, and gives it half a second to answer.
  2. Choose llama.cpp. Its address is filled in, with Found running on this machine. under it when fxbot found it. If llama-server runs on another port, change it, such as to http://localhost:8080/v1 for an older build.
  3. Leave API key, if the server needs one empty, unless you started it with --api-key. Set Context window, and choose Add. fxbot checks the server first.
  4. Pick the model. Tick Make it the default for new bots to start each new bot on it.
Field What to put there
Address llama-server’s, with its /v1. The usual one, http://localhost:9931/v1, is filled in.
API key, if the server needs one One of its --api-key keys, if you gave any. It goes to fxbot’s server once, sealed, and never comes back to the page.
Context window The context llama-server runs with: its n_ctx, or the -c you gave it. fxbot can’t read it from llama-server’s list. It applies to every model on this provider.
Name Optional. It’s llama.cpp when you leave it empty.

When fxbot runs from a checkout, with npm run dev, it runs on the machine itself, and the usual address works as it is.

A release install, the one setup.sh makes, runs fxbot in a container. There, localhost is the container, not your machine, so fxbot doesn’t find llama-server, and the usual address doesn’t reach it. fxbot takes an HTTPS address, or plain HTTP only on localhost. A server on the machine is plain HTTP from inside the container, which fxbot can’t use yet, and the card says so. The same goes for a server on another machine.

Put llama-server behind HTTPS, and give fxbot that address. With Tailscale, and HTTPS turned on for your tailnet, serve it on a port of its own, such as 8443:

Terminal window
sudo tailscale serve --bg --https=8443 9931

Then give fxbot https://<machine>.<tailnet>.ts.net:8443/v1 as the Address. fxbot’s own tailscale serve, on port 443, stays as it is (Reach fxbot from other devices).

fxbot lists the model from llama-server’s own list, /v1/models, by its id: the model file’s path, or its --alias. Newer builds say whether it reads images. The list gives no context window or tool use that fxbot reads, so set Context window yourself.

  • “The key check couldn’t reach localhost:9931.” llama-server isn’t running there. Older builds listen on 8080: put its port in the address. In a release install, that’s the container’s own localhost: see In a release install.
  • “llama.cpp rejected this key (HTTP 401): it’s wrong, or it was deleted.” llama-server was started with --api-key, and the key you gave isn’t one of them, or you gave none.
  • “baseUrl must use https, or http on localhost.” The address is plain HTTP to somewhere other than localhost or 127.0.0.1. Give an HTTPS address.
  • The model writes tool calls as text, or calls none. Start llama-server with --jinja. If it still doesn’t, the model may need --chat-template-file, as llama.cpp’s function calling page says.