← Templates · Dev Tools
VL
vLLM
CommunityHigh-throughput, memory-efficient inference and serving engine for large language models, with an OpenAI-compatible API.
| Runs | vllm/vllm-openai:v0.31.0 |
|---|---|
| Stack | python |
| Provisioned for it | Nothing, it's self-contained |
| Verified | Not yet |
Why deploy on Railyard
Railyard runs it standalone on a server you already pay for — no addons to provision, with zero-downtime deploys and scaling still included.
Deploying vLLM is one piece of a bigger stack you can keep building — add a database, a worker, a cache, or another app, all on the same servers you already pay for.
Good for
- High-throughput self-hosted inference for an open-weight model
- An OpenAI-compatible endpoint for a model too large or too frequently used to run well on Ollama
- Serving a fine-tuned model behind your own API
Good to know
- Needs a real NVIDIA GPU — there's no practical CPU fallback for serving most models
- Change the placeholder --model in the start command to the model you actually want served; first boot downloads it from Hugging Face (add HF_TOKEN for gated models)