vLLM serves open models from Hugging Face on your own machine’s GPU, fast, and to many requests at once. It’s free and open source, and runs on Linux. Your own hardware does the work.
Start the server
Section titled “Start the server”-
Install vLLM, as its quickstart says.
-
Serve a model, with tool calls turned on:
Terminal window vllm serve Qwen/Qwen2.5-1.5B-Instruct --enable-auto-tool-choice --tool-call-parser hermes
--enable-auto-tool-choicelets the model call tools when it chooses to, which a bot’s tools need. vLLM leaves it off unless you give it.--tool-call-parsersays how the model writes its tool calls, which differs by model:hermesfor Qwen2.5 and Hermes,llama3_jsonfor Llama 3.1 and 3.2,mistralfor Mistral, andopenaifor gpt-oss. vLLM’s tool calling page lists the rest, and the models that need a--chat-templatetoo.--max-model-lensets the context. Without it, vLLM takes the model’s own.--served-model-namegives the model the id fxbot shows. Without it, the id is the model’s name, as you gave it.--api-key, orVLLM_API_KEY, makes it ask for a key. Without either, it takes none.
vLLM listens on port 8000, and serves an OpenAI-compatible API under /v1.
Add it to fxbot
Section titled “Add it to fxbot”- In Settings › Providers, choose Add provider. Under On your machine, vLLM says Found here when it’s running at its usual address with no key. fxbot asks each server there for its model list when the page opens, and gives it half a second to answer.
- Choose vLLM. Its address is filled in, with Found running on this machine. under it when fxbot found it.
- Leave API key, if the server needs one empty, unless you gave vLLM a key. Choose Add. fxbot checks the server first.
- Pick the model. Tick Make it the default for new bots to start each new bot on it.
| Field | What to put there |
|---|---|
| Address | vLLM’s, with its /v1. The usual one, http://localhost:8000/v1, is filled in. |
| API key, if the server needs one | The key you gave --api-key or VLLM_API_KEY, if any. It goes to fxbot’s server once, sealed, and never comes back to the page. |
| Context window | Optional. vLLM reports each model’s, so leave it empty unless that’s wrong. A number here overrides what vLLM reports, for every model on this provider. |
| Name | Optional. It’s vLLM when you leave it empty. |
When fxbot runs from a checkout, with npm run dev, it runs on the machine itself, and the usual address works as it is.
In a release install
Section titled “In a release install”A release install, the one setup.sh makes, runs fxbot in a container. There, localhost is the container, not your machine, so fxbot doesn’t find vLLM, and the usual address doesn’t reach it. fxbot takes an HTTPS address, or plain HTTP only on localhost. A server on the machine is plain HTTP from inside the container, which fxbot can’t use yet, and the card says so. The same goes for a server on another machine.
Put vLLM behind HTTPS, and give fxbot that address. With Tailscale, and HTTPS turned on for your tailnet, serve it on a port of its own, such as 8443:
sudo tailscale serve --bg --https=8443 8000Then give fxbot https://<machine>.<tailnet>.ts.net:8443/v1 as the Address. fxbot’s own tailscale serve, on port 443, stays as it is (Reach fxbot from other devices).
Its models
Section titled “Its models”fxbot lists the models from vLLM’s own list, /v1/models: the one it serves, by its id, with the context window vLLM runs it with, its --max-model-len. vLLM’s list doesn’t say whether a model uses tools or reads images.
When it fails
Section titled “When it fails”- “The key check couldn’t reach localhost:8000.” vLLM isn’t running there, or is still loading the model. Wait for it, or give the address it runs at. In a release install, that’s the container’s own localhost: see In a release install.
- “vLLM rejected this key (HTTP 401): it’s wrong, or it was deleted.” vLLM was started with a key, and the one you gave isn’t it, or you gave none.
- “baseUrl must use https, or http on localhost.” The address is plain HTTP to somewhere other than
localhostor127.0.0.1. Give an HTTPS address. - A chat fails at once, with an error from vLLM about tool choice. vLLM was started without
--enable-auto-tool-choiceand a--tool-call-parser. A bot’s tools need both: start it with them. - The model writes tool calls as text. The parser doesn’t match the model. Pick the one vLLM’s tool calling page names for it.