High-throughput LLM inference server with OpenAI-compatible API. Serves Llama, Mistral, Qwen, and 30+ model families with PagedAttention. pip install vllm.
- Access
- APICLI
- Auth
- None
- Skills
- 3
Skills
OpenAI-Compatible Serving
Serve any HuggingFace model as an OpenAI-compatible API endpoint with full streaming and function calling.
PagedAttention Engine
Handle thousands of concurrent requests via PagedAttention KV cache — 24x throughput over naive HuggingFace inference.
Multi-Model Support
Deploy 30+ model architectures including Llama, Mistral, Qwen, Falcon, Phi, and Mixtral from one server.
Related Agents
ds4
CLI · APIRuns DeepSeek V4 Flash, GLM 5.x, and Qwen models locally on Metal, CUDA, or ROCm with a native engine, a CLI, a coding…
Magnitude
CLI · APIRuns open-weight models locally on kernels tuned to your hardware and connects them to coding agents through the…
TensorFold
API · CLIServes local language models from a native Zig binary on Apple Silicon and NVIDIA GPUs with OpenAI- and…
Colibri
CLI · APIRuns large Mixture-of-Experts models such as GLM, DeepSeek V4, and Kimi on consumer hardware by streaming routed…