# vLLM vLLM is an open-source inference and serving framework for large language models (LLMs). This section describes how to run vLLM on Qualcomm Cloud AI using container images, and links to feature documentation and reference material. ## Architecture vLLM serving stack on Qualcomm Cloud AI accelerators. ## Highlights - [Prebuilt Docker image](https://docs.qualcomm.com/doc/80-99100-3/topic/run.html) - Run vLLM with a prebuilt Qualcomm Cloud AI Docker image. - [Disaggregated serving](https://docs.qualcomm.com/doc/80-99100-3/topic/Disaggregated-Serving.html) - Scale prefill and decode independently for higher throughput and better utilization. - [Compute context length (CCL)](https://docs.qualcomm.com/doc/80-99100-3/topic/CCL.html) - Accelerate long-context prompts by switching compute context lengths at runtime. - [Speculative decoding](https://docs.qualcomm.com/doc/80-99100-3/topic/Speculative-Decoding.html) and [prefix caching](https://docs.qualcomm.com/doc/80-99100-3/topic/Prefix-Caching.html) - Reduce latency with draft-model proposals and prefix reuse. - [On-device sampling](https://docs.qualcomm.com/doc/80-99100-3/topic/On-Device-Sampling.html) - Run sampling on the accelerator to lower host overhead. - [Quantization](https://docs.qualcomm.com/doc/80-99100-3/topic/Quantized-model.html) - MXFP6 weight compression and MXINT8 activations for higher throughput and lower memory. - [Tool call parsing](https://docs.qualcomm.com/doc/80-99100-3/topic/Tool-Call-Parsing.html) and [guided decoding](https://docs.qualcomm.com/doc/80-99100-3/topic/Guided-Decoding.html) - Structured outputs and constrained decoding. - [Multimodality](https://docs.qualcomm.com/doc/80-99100-3/topic/Multimodality.html) and [LoRAX](https://docs.qualcomm.com/doc/80-99100-3/topic/LoRAX.html) - Vision-language models and adapter-based personalization. See [supported features](https://docs.qualcomm.com/doc/80-99100-3/topic/features.html) for a complete list. ## Quickstart Run your first model with vLLM on Qualcomm Cloud AI accelerators: 1. Do the following to start the server: > > > docker run --rm -it --network host \ > --device /dev/accel/ \ > --shm-size=2gb \ > ghcr.io/quic/cloud_ai_inference_vllm:1.21.2.0 \ > --host 127.0.0.1 \ > --port 8000 \ > --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ > --max-model-len 256 \ > --max-num-seq 16 \ > --max-seq-len-to-capture 128 \ > --quantization mxfp6 \ > --kv-cache-dtype mxint8 > Copy to clipboard 2. Do the following to send a chat request: curl http://127.0.0.1:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "Explain vLLM in one sentence." } ], "temperature": 0.7, "max_tokens": 128 }' Copy to clipboard ## Next steps - [Run vLLM](https://docs.qualcomm.com/doc/80-99100-3/topic/run.html): Run vLLM using a prebuilt Qualcomm Cloud AI Docker image. - [Benchmarking](https://docs.qualcomm.com/doc/80-99100-3/topic/benchmarking.html): Validate the vLLM server and measure performance. - [Supported features](https://docs.qualcomm.com/doc/80-99100-3/topic/features.html): See a list of vLLM features model coverage, serving capabilities, and performance optimizations. - [Build vLLM (optional)](https://docs.qualcomm.com/doc/80-99100-3/topic/build.html): Add backend support to the vLLM open-source library to simplify the creation of OpenAI-compatible web endpoints. Last Published: Aug 25, 2026 [Previous Topic Serve the model](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/index_Model-Serving.md) [Next Topic Run vLLM](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/run.md)