Skip to content

vLLM

Copy all docs

vLLM serves open models from Hugging Face on your own machine’s GPU, fast, and to many requests at once. It’s free and open source, and runs on Linux. Your own hardware does the work.

  1. Install vLLM, as its quickstart says.

  2. Serve a model, with tool calls turned on:

    Terminal window
    vllm serve Qwen/Qwen2.5-1.5B-Instruct --enable-auto-tool-choice --tool-call-parser hermes
  • --enable-auto-tool-choice lets the model call tools when it chooses to, which a bot’s tools need. vLLM leaves it off unless you give it.
  • --tool-call-parser says how the model writes its tool calls, which differs by model: hermes for Qwen2.5 and Hermes, llama3_json for Llama 3.1 and 3.2, mistral for Mistral, and openai for gpt-oss. vLLM’s tool calling page lists the rest, and the models that need a --chat-template too.
  • --max-model-len sets the context. Without it, vLLM takes the model’s own.
  • --served-model-name gives the model the id fxbot shows. Without it, the id is the model’s name, as you gave it.
  • --api-key, or VLLM_API_KEY, makes it ask for a key. Without either, it takes none.

vLLM listens on port 8000, and serves an OpenAI-compatible API under /v1.

  1. In Settings › Providers, choose Add provider. Under On your machine, vLLM says Found here when it’s running at its usual address with no key. fxbot asks each server there for its model list when the page opens, and gives it half a second to answer.
  2. Choose vLLM. Its address is filled in, with Found running on this machine. under it when fxbot found it.
  3. Leave API key, if the server needs one empty, unless you gave vLLM a key. Choose Add. fxbot checks the server first.
  4. Pick the model. Tick Make it the default for new bots to start each new bot on it.
Field What to put there
Address vLLM’s, with its /v1. The usual one, http://localhost:8000/v1, is filled in.
API key, if the server needs one The key you gave --api-key or VLLM_API_KEY, if any. It goes to fxbot’s server once, sealed, and never comes back to the page.
Context window Optional. vLLM reports each model’s, so leave it empty unless that’s wrong. A number here overrides what vLLM reports, for every model on this provider.
Name Optional. It’s vLLM when you leave it empty.

When fxbot runs from a checkout, with npm run dev, it runs on the machine itself, and the usual address works as it is.

A release install, the one setup.sh makes, runs fxbot in a container. There, localhost is the container, not your machine, so fxbot doesn’t find vLLM, and the usual address doesn’t reach it. fxbot takes an HTTPS address, or plain HTTP only on localhost. A server on the machine is plain HTTP from inside the container, which fxbot can’t use yet, and the card says so. The same goes for a server on another machine.

Put vLLM behind HTTPS, and give fxbot that address. With Tailscale, and HTTPS turned on for your tailnet, serve it on a port of its own, such as 8443:

Terminal window
sudo tailscale serve --bg --https=8443 8000

Then give fxbot https://<machine>.<tailnet>.ts.net:8443/v1 as the Address. fxbot’s own tailscale serve, on port 443, stays as it is (Reach fxbot from other devices).

fxbot lists the models from vLLM’s own list, /v1/models: the one it serves, by its id, with the context window vLLM runs it with, its --max-model-len. vLLM’s list doesn’t say whether a model uses tools or reads images.

  • “The key check couldn’t reach localhost:8000.” vLLM isn’t running there, or is still loading the model. Wait for it, or give the address it runs at. In a release install, that’s the container’s own localhost: see In a release install.
  • “vLLM rejected this key (HTTP 401): it’s wrong, or it was deleted.” vLLM was started with a key, and the one you gave isn’t it, or you gave none.
  • “baseUrl must use https, or http on localhost.” The address is plain HTTP to somewhere other than localhost or 127.0.0.1. Give an HTTPS address.
  • A chat fails at once, with an error from vLLM about tool choice. vLLM was started without --enable-auto-tool-choice and a --tool-call-parser. A bot’s tools need both: start it with them.
  • The model writes tool calls as text. The parser doesn’t match the model. Pick the one vLLM’s tool calling page names for it.