LiteLLM’s proxy is a server you run yourself, in front of other providers: clouds, labs, and servers on your own machine, such as Ollama or vLLM. It gives them all one OpenAI-compatible API, with keys of its own. It’s free and open source. The providers behind it bill as they do.
Start the server
Section titled “Start the server”-
Install it, as LiteLLM’s quick start says:
Terminal window uv tool install 'litellm[proxy]' -
Write a
config.yamlthat lists its models undermodel_list: for each one, themodel_namefxbot will show, and underlitellm_params, the provider’s model and key. LiteLLM’s config page shows how. -
Start it:
Terminal window litellm --config config.yaml
It listens on port 4000, with an OpenAI-compatible API under /v1. Whether a model calls tools depends on the model and the provider behind it.
Its keys:
- No master key: the proxy takes requests with no key.
- A master key,
master_keyundergeneral_settingsinconfig.yaml, orLITELLM_MASTER_KEY, starts withsk-. Every request then needs a key: the master key, or a virtual key. - A virtual key is made with the master key, at the proxy’s
/key/generate. Virtual keys need a Postgres database of LiteLLM’s own, set withDATABASE_URL. One made for fxbot can be removed without touching the rest.
Add it to fxbot
Section titled “Add it to fxbot”- In Settings › Providers, choose Add provider. Under On your machine, LiteLLM says Found here when the proxy is running at its usual address with no master key. fxbot asks each server there for its model list when the page opens, without a key, and gives it half a second to answer. With a master key, the tile doesn’t say so, and the address is filled in all the same.
- Choose LiteLLM. Its address is filled in.
- Paste a virtual key, or the master key, and choose Add. fxbot checks it with the proxy first.
- Pick a model. Tick Make it the default for new bots to start each new bot on it.
| Field | What to put there |
|---|---|
| Address | The proxy’s, with its /v1. The usual one, http://localhost:4000/v1, is filled in. |
| API key, if the server needs one | A virtual key, or the master key. Leave it empty when the proxy has no master key. It goes to fxbot’s server once, sealed, and never comes back to the page. |
| Context window | The proxy’s list doesn’t say it. A number here applies to every model behind the proxy, so give the smallest of those you’ll use. |
| Name | Optional. It’s LiteLLM when you leave it empty. |
When fxbot runs from a checkout, with npm run dev, it runs on the machine itself, and the usual address works as it is.
In a release install
Section titled “In a release install”A release install, the one setup.sh makes, runs fxbot in a container. There, localhost is the container, not your machine, so fxbot doesn’t find the proxy, and the usual address doesn’t reach it. fxbot takes an HTTPS address, or plain HTTP only on localhost. A server on the machine is plain HTTP from inside the container, which fxbot can’t use yet, and the card says so. The same goes for a server on another machine.
Put the proxy behind HTTPS, and give fxbot that address. With Tailscale, and HTTPS turned on for your tailnet, serve it on a port of its own, such as 8443:
sudo tailscale serve --bg --https=8443 4000Then give fxbot https://<machine>.<tailnet>.ts.net:8443/v1 as the Address. fxbot’s own tailscale serve, on port 443, stays as it is (Reach fxbot from other devices).
Its models
Section titled “Its models”fxbot lists the models from the proxy’s own list, /v1/models, by their ids: each model_name in its config. The list gives ids only, with no context window, tool use, or images.
When it fails
Section titled “When it fails”- “The key check couldn’t reach localhost:4000.” The proxy isn’t running there. Start it, or give the address it runs at. In a release install, that’s the container’s own localhost: see In a release install.
- “LiteLLM rejected this key (HTTP 401): it’s wrong, or it was deleted.” The key isn’t one the proxy knows, or the proxy has a master key and you gave none.
- “baseUrl must use https, or http on localhost.” The address is plain HTTP to somewhere other than
localhostor127.0.0.1. Give an HTTPS address. - A chat fails on one model, and not the others. The provider behind that model is failing, such as with a wrong key in
config.yaml. The proxy’s log says why.