Skip to main content

Ctrl+K

You are viewing the latest developer preview docs. Click here to view docs for the latest stable release.

Getting Started

Installation
Installation with ROCm
Installation with OpenVINO
Installation with CPU
Installation with Neuron
Installation with TPU
Installation with XPU
Quickstart
Debugging Tips
Examples

Serving

OpenAI Compatible Server
Deploying with Docker
Distributed Inference and Serving
Production Metrics
Environment Variables
Usage Stats Collection
Integrations
Loading Models with CoreWeave’s Tensorizer
Frequently Asked Questions

Models

Supported Models
Adding a New Model
Engine Arguments
Using LoRA adapters
Using VLMs
Speculative decoding in vLLM
Performance and Tuning

Quantization

Supported Hardware for Quantization Kernels
AutoAWQ
FP8
FP8 E5M2 KV Cache
FP8 E4M3 KV Cache

Automatic Prefix Caching

Introduction
Implementation

Developer Documentation

Sampling Parameters
Offline Inference
- LLM Class
- LLM Inputs
vLLM Engine
- LLMEngine
- AsyncLLMEngine
vLLM Paged Attention
Input Processing
- Input Processing Pipeline
Multi-Modality
- Adding a New Multimodal Model
Dockerfile

Community

vLLM Meetups
Sponsors

Search

Ctrl+K

By the vLLM Team

© Copyright 2024, vLLM Team.