vLLM
Verifiedby vLLM Project
High-throughput LLM inference server with OpenAI-compatible API. Serves Llama, Mistral, Qwen, and 30+ model families with PagedAttention. pip install vllm.
- Access
- APICLI
- Auth
- None
- Skills
- 3
Skills
OpenAI-Compatible Serving
Serve any HuggingFace model as an OpenAI-compatible API endpoint with full streaming and function calling.
PagedAttention Engine
Handle thousands of concurrent requests via PagedAttention KV cache — 24x throughput over naive HuggingFace inference.
Multi-Model Support
Deploy 30+ model architectures including Llama, Mistral, Qwen, Falcon, Phi, and Mixtral from one server.
Related Agents
ds4
by Salvatore Sanfilippo (antirez)
Runs DeepSeek V4 Flash, GLM 5.x, and Qwen models locally on Metal, CUDA, or ROCm with a native engine, a CLI, a coding…
Magnitude
by Magnitude
Runs open-weight models locally on kernels tuned to your hardware and connects them to coding agents through the…
TensorFold
by ashhart
Serves local language models from a native Zig binary on Apple Silicon and NVIDIA GPUs with OpenAI- and…
Colibri
by JustVugg
Runs large Mixture-of-Experts models such as GLM, DeepSeek V4, and Kimi on consumer hardware by streaming routed…