# Run vLLM

This page shows how to run vLLM using a prebuilt Qualcomm Cloud AI Docker image.

## Pull the image

docker pull ghcr.io/quic/cloud_ai_inference_vllm:1.21.2.0
    Copy to clipboard

## Start the server

docker run --rm -it --network host \
       --workdir /workspace \
       --device /dev/accel/ \
       --shm-size=2gb \
       --mount type=bind,source=$PWD,target=/workspace \
       --mount type=bind,source=$HOME/.cache,target=/cache \
       -e HF_HOME=/cache/huggingface \
       -e QEFF_HOME=/cache/qeff_models \
       ghcr.io/quic/cloud_ai_inference_vllm:1.21.2.0 \
       --host 127.0.0.1 \
       --port 8000 \
       --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
       --max-model-len 256 \
       --max-num-seq 16 \
       --max-seq-len-to-capture 128 \
       --quantization mxfp6 \
       --kv-cache-dtype mxint8
    Copy to clipboard

Note

This example mounts a host workspace and maps cache directories so model weights and QPC artifacts are stored on the host rather than inside the container.

This prevents losing them when the container exits and avoids recompiling on every restart. The first run may take significant time (Hugging Face download, ONNX export, QPC compilation), but subsequent runs are much faster when the caches are reused.

Cache locations:

- Hugging Face model weights:

> 
> 
> $HOME/.cache/huggingface
>         Copy to clipboard
- QPCs and intermediate artifacts:

> 
> 
> $HOME/.cache/qeff_models
>         Copy to clipboard

### Hugging Face authentication (HF\_TOKEN)

Some models (for example, gated or private models) require authentication.

- Do the following to provide your HF\_TOKEN:

> 
> 
> -e HF_TOKEN=<your_huggingface_token>
>         Copy to clipboard

HF\_TOKEN isn’t required for fully public models like TinyLlama.

## Run on bare metal

For bare metal, follow the vLLM installation from source section of [vLLM installation](https://docs.qualcomm.com/doc/80-99100-3/topic/index_vLLM-Serving.html#reference-to-vllm) - Qualcomm Cloud AI Documentation.

vLLM provides capabilities to start a FastAPI server to run LLM
inference. Here is an example to use qaic backend (that is, use the Qualcomm Cloud AI
accelerators for inference).

# Need to increase max open files to serve multiple requests
    ulimit -n 1048576
    
    # Need to configure thread parallelism to avoid unnecessary CPU contention
    export OMP_NUM_THREADS=8
    
    # Start the server
    python3 -m vllm.entrypoints.api_server --host 127.0.0.1 --port 8000 --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --max-model-len 256 --max-num-seq 16 --max-seq_len-to-capture 128 --device qaic --block-size 32 --quantization mxfp6 --kv-cache-dtype mxint8
    
    # Client request
    python3 examples/api_client.py --host 127.0.0.1 --port 8000 --prompt "My name is" --stream
    Copy to clipboard

Similarly, an OpenAI compatible server can be invoked as follows

# Need to increase max open files to serve multiple requests
    ulimit -n 1048576
    
    # Need to configure thread parallelism to avoid unnecessary CPU contention
    export OMP_NUM_THREADS=8
    
    # Start the server
    python3 -m vllm.entrypoints.openai.api_server --host 127.0.0.1 --port 8000 --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --max-model-len 256 --max-num-seq 16 --max-seq_len-to-capture 128 --device qaic --block-size 32 --quantization mxfp6 --kv-cache-dtype mxint8
    
    # Client request
    python3 examples/openai_chat_completion_client.py
    Copy to clipboard

Last Published: Aug 25, 2026

[Previous Topic
vLLM](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/index_vLLM-Serving.md) [Next Topic
Benchmarking](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/benchmarking.md)