Skip to main content
Ctrl+K
vLLM - Home

Getting Started

  • Installation
  • Installation with ROCm
  • Installation with Neuron
  • Installation with CPU
  • Quickstart
  • Examples
    • API Client
    • Aqlm Example
    • Gradio OpenAI Chatbot Webserver
    • Gradio Webserver
    • Llava Example
    • LLM Engine Example
    • MultiLoRA Inference
    • Offline Bench
    • Offline Inference
    • Offline Inference 24
    • Offline Inference Arctic
    • Offline Inference Distributed
    • Offline Inference Embedding
    • Offline Inference Neuron
    • Offline Inference Sparse
    • Offline Inference With Prefix
    • Offline Profile
    • OpenAI Chat Completion Client
    • OpenAI Completion Client
    • OpenAI Embedding Client
    • Save Sharded State
    • Tensorize vLLM Model

Offline Inference

  • LLM Class
  • Sampling Parameters

Serving

  • OpenAI Compatible Server
  • Deploying with Docker
  • Distributed Inference and Serving
  • Production Metrics
  • Environment Variables
  • Usage Stats Collection
  • Integrations
    • Deploying and scaling up with SkyPilot
    • Deploying with KServe
    • Deploying with NVIDIA Triton
    • Deploying with BentoML
    • Deploying with LWS
    • Serving with Langchain

Models

  • Supported Models
  • Adding a New Model
  • Engine Arguments
  • Using LoRA adapters
  • Performance and Tuning

Quantization

  • AutoAWQ
  • FP8 E5M2 KV Cache
  • FP8 E4M3 KV Cache

Developer Documentation

  • vLLM Engine
    • LLMEngine
    • AsyncLLMEngine
  • vLLM Paged Attention
  • Dockerfile

Community

  • vLLM Meetups
  • .rst

Integrations

Integrations#

  • Deploying and scaling up with SkyPilot
  • Deploying with KServe
  • Deploying with NVIDIA Triton
  • Deploying with BentoML
  • Deploying with LWS
  • Serving with Langchain

previous

Usage Stats Collection

next

Deploying and scaling up with SkyPilot

By the vLLM Team

© Copyright 2024, vLLM Team.