Skip to content

vLLM: High-Throughput Distributed LLM Serving Engine

Datacenter-grade LLM serving framework powered by PagedAttention, continuous token-level batching, chunked prefill, and multi-GPU tensor parallelism.

36.8kApache-2.0PythonUpdated 2 days ago
GitHub

Hardware Pre-Flight Check

Verify your system meets runtime requirements before setup

Verified Specs
System RAM16 GB MinRec: 32 GB
GPU VRAM12 GB MinRec: 24 GB
Free Storage~30 GBFast SSD Recommended
Accelerators
NVIDIA CUDAAMD ROCmIntel GaudiAWS Neuron

Will vLLM run on your PC?

We can automatically scan your local GPU and System RAM right from the browser to see if your hardware meets the requirements for this repository. No downloads required.

Cloud GPU Infrastructure

Deploy vLLM on Cloud GPUs

Need high-performance compute without buying an enterprise GPU? Rent on-demand cards instantly.

RunPodMost Popular
$0.29/hrstarting rate

Instant cloud GPUs with 1-click vLLM, Ollama, and PyTorch templates. RTX 4090 to 8x H100.

Best for: Ollama, vLLM & Rapid LLM Serving
Vast.aiLowest Price
$0.18/hrstarting rate

Decentralized GPU marketplace with aggressive per-hour rates for consumer & datacenter cards.

Best for: Budget LoRA Fine-Tuning & Batch Inference
Lambda LabsEnterprise Performance
$0.50/hrstarting rate

High-speed networked enterprise clusters with low latency and dedicated NVMe storage.

Best for: Multi-GPU Training & Production Serving

Rates verified weekly. Transparent disclosure: We may receive an affiliate commission from partner signups at no extra cost to you.

3-Tier Audience Playbook

Actionable guidance tailored to your technical workflow

3 Experience Levels
Beginner Quickstart

Install via pip in a clean Python 3.10+ virtualenv with CUDA 12.1+ drivers. Run a single-GPU server with a quantized AWQ checkpoint.

Intermediate Production

Deploy as a production microservice with Docker. Point external web apps or reverse proxies to the OpenAI-compatible /v1 endpoints.

Advanced Scaling

Configure distributed multi-GPU serving with --tensor-parallel-size, optimize KV cache block size, and enable speculative decoding with draft models.

What is vLLM?

vLLM is a high-performance open-source inference and serving engine designed for LLMs. Created at UC Berkeley, it introduced PagedAttention, an algorithm that treats the Key-Value (KV) cache as virtual memory pages, reducing memory waste from over 60% down to under 4%.

This architectural breakthrough allows vLLM to dynamically batch multiple incoming generation requests at the token level, delivering enterprise-scale throughput on modern GPUs.

Key Architectural Capabilities

  • PagedAttention: Virtual memory allocation for KV cache tensors eliminates fragmentation and enables efficient memory sharing across parallel requests.
  • Continuous Batching: Requests are scheduled at token iterations rather than waiting for entire sequences to finish, maximizing GPU compute utilization.
  • Chunked Prefill: Long prompt prefill computations are divided into smaller chunks and interleaved with decoding steps to protect tail latency.
  • Tensor Parallelism: Seamlessly shard models across 2, 4, or 8 GPUs on single or multi-node clusters.
  • Quantization Support: Native FP8, AWQ, GPTQ, and Marlin kernels for accelerated inference with lower memory overhead.

Installation & Setup

vLLM requires Linux with an NVIDIA GPU (Compute Capability 7.0+) or AMD ROCm support:

# Recommended: Create an isolated environment
python3 -m venv vllm-env
source vllm-env/bin/activate

# Install vLLM with CUDA 12 support
pip install --upgrade pip
pip install vllm

What are the core runtime flags and configuration parameters for vLLM?

Flag / Parameter Type Signature Default Value Description & Production Validation Rules
--model string (Required) HuggingFace model repo ID or local checkpoint path with tokenizer.
--port integer 8000 Port to bind the OpenAI-compatible HTTP REST server. Range: 1 to 65535.
--gpu-memory-utilization float [0.0–1.0] 0.90 Fraction of total GPU VRAM reserved for model weights and dynamic KV-cache.
--max-model-len integer Model Config Maximum context window (prompt + completion tokens). Caps KV-cache allocation.
--tensor-parallel-size integer 1 Number of GPUs to shard weights across using Megatron-LM tensor parallelism.
--quantization enum None Quantization backend: awq, gptq, fp8, bitsandbytes, or marlin.
--enforce-eager boolean false Disables CUDA graph capture; saves ~1.5 GB VRAM at the cost of slight latency.

Production Deployment Recipes

1. Launch OpenAI-Compatible Model Server

python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --enforce-eager

2. Multi-GPU Tensor Parallelism (e.g. 2x RTX 4090 / 2x A100)

python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --quantization awq \
  --port 8000

3. Docker Container Deployment

docker run --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.90

Objective Alternatives & Tradeoffs Matrix

Head-to-head comparison without vendor bias or dismissal

TGI (Text Generation Inference)• KV cache memory management and throughput scaling
Project Link↗
✓ Choose vLLM if:

You need maximum token throughput, PagedAttention memory efficiency, and broad multi-architecture support.

⚖ Choose TGI (Text Generation Inference) if:

You are heavily standardized on HuggingFace Hub workflows and specific enterprise SLA deployments.

Ollama• Datacenter multi-concurrency server vs developer workstation utility
Compare Guide→
✓ Choose vLLM if:

You are serving production APIs with 10+ concurrent requests per second.

⚖ Choose Ollama if:

You want a lightweight local daemon on your personal laptop.

Common Production Pitfalls & Gotchas

Battle-tested solutions for frequent setup errors and bottlenecks

!

CUDA Out of Memory on startup before processing any requests

Tested Fix:vLLM pre-allocates GPU memory for the KV cache. Tune '--gpu-memory-utilization 0.85' or reduce '--max-model-len 4096' to fit in your GPU memory.
!

Tensor parallelism communication hangs on multi-GPU systems

Tested Fix:Ensure NCCL environment variables are properly set (NCCL_DEBUG=INFO) and disable peer-to-peer if PCIe switches do not support direct P2P.
!

High first-token latency during long prompt processing

Tested Fix:Enable chunked prefill with '--enable-chunked-prefill' to prevent long prompts from stalling token generation for active streams.

Comments are powered by GitHub Discussions and will appear here once connected.