Skip to content

LiteLLM

Copy all docs

LiteLLM’s proxy is a server you run yourself, in front of other providers: clouds, labs, and servers on your own machine, such as Ollama or vLLM. It gives them all one OpenAI-compatible API, with keys of its own. It’s free and open source. The providers behind it bill as they do.

  1. Install it, as LiteLLM’s quick start says:

    Terminal window
    uv tool install 'litellm[proxy]'
  2. Write a config.yaml that lists its models under model_list: for each one, the model_name fxbot will show, and under litellm_params, the provider’s model and key. LiteLLM’s config page shows how.

  3. Start it:

    Terminal window
    litellm --config config.yaml

It listens on port 4000, with an OpenAI-compatible API under /v1. Whether a model calls tools depends on the model and the provider behind it.

Its keys:

  • No master key: the proxy takes requests with no key.
  • A master key, master_key under general_settings in config.yaml, or LITELLM_MASTER_KEY, starts with sk-. Every request then needs a key: the master key, or a virtual key.
  • A virtual key is made with the master key, at the proxy’s /key/generate. Virtual keys need a Postgres database of LiteLLM’s own, set with DATABASE_URL. One made for fxbot can be removed without touching the rest.
  1. In Settings › Providers, choose Add provider. Under On your machine, LiteLLM says Found here when the proxy is running at its usual address with no master key. fxbot asks each server there for its model list when the page opens, without a key, and gives it half a second to answer. With a master key, the tile doesn’t say so, and the address is filled in all the same.
  2. Choose LiteLLM. Its address is filled in.
  3. Paste a virtual key, or the master key, and choose Add. fxbot checks it with the proxy first.
  4. Pick a model. Tick Make it the default for new bots to start each new bot on it.
Field What to put there
Address The proxy’s, with its /v1. The usual one, http://localhost:4000/v1, is filled in.
API key, if the server needs one A virtual key, or the master key. Leave it empty when the proxy has no master key. It goes to fxbot’s server once, sealed, and never comes back to the page.
Context window The proxy’s list doesn’t say it. A number here applies to every model behind the proxy, so give the smallest of those you’ll use.
Name Optional. It’s LiteLLM when you leave it empty.

When fxbot runs from a checkout, with npm run dev, it runs on the machine itself, and the usual address works as it is.

A release install, the one setup.sh makes, runs fxbot in a container. There, localhost is the container, not your machine, so fxbot doesn’t find the proxy, and the usual address doesn’t reach it. fxbot takes an HTTPS address, or plain HTTP only on localhost. A server on the machine is plain HTTP from inside the container, which fxbot can’t use yet, and the card says so. The same goes for a server on another machine.

Put the proxy behind HTTPS, and give fxbot that address. With Tailscale, and HTTPS turned on for your tailnet, serve it on a port of its own, such as 8443:

Terminal window
sudo tailscale serve --bg --https=8443 4000

Then give fxbot https://<machine>.<tailnet>.ts.net:8443/v1 as the Address. fxbot’s own tailscale serve, on port 443, stays as it is (Reach fxbot from other devices).

fxbot lists the models from the proxy’s own list, /v1/models, by their ids: each model_name in its config. The list gives ids only, with no context window, tool use, or images.

  • “The key check couldn’t reach localhost:4000.” The proxy isn’t running there. Start it, or give the address it runs at. In a release install, that’s the container’s own localhost: see In a release install.
  • “LiteLLM rejected this key (HTTP 401): it’s wrong, or it was deleted.” The key isn’t one the proxy knows, or the proxy has a master key and you gave none.
  • “baseUrl must use https, or http on localhost.” The address is plain HTTP to somewhere other than localhost or 127.0.0.1. Give an HTTPS address.
  • A chat fails on one model, and not the others. The provider behind that model is failing, such as with a wrong key in config.yaml. The proxy’s log says why.