# vLLM Backend

The vLLM (0.10.1.1) backend for Triton is a Python-based backend designed to
run [supported
models](https://github.com/quic/efficient-transformers/blob/release/v1.21.0/docs/source/validate.md)
on the vLLM AsyncEngine.

See [vLLM](https://docs.qualcomm.com/doc/80-99100-3/topic/index_vLLM-Serving.html#reference-to-vllm) for more information about model.json configuration parameters, environment variables and benchmarking support.

## Start vLLM Models

Sample model repository for TinyLlama model is generated at
`"/opt/qti-aic/aic-triton-model-repositories/vllm_model"` while
building Triton docker with triton\_model\_repo application using
[Docker](https://docs.qualcomm.com/doc/80-99100-3/topic/index_Docker.html#reference-to-docker). You can use this as
is or change the model by changing model value that’s passed to the
vLLM AsyncEngine. Modify `model.json` as needed.

`model.json` represents a key-value dictionary that’s fed to the vLLM’s AsyncEngine. Modify the model.json as needed.

model.json sample parameters.

{
        "model": "model_name",
        "device_group": [0,1,2,3,4], # device_id for execution
        "max_num_seqs": <decode_bsz>, # Decode batch size
        "max_model_len": <ctx_len>, # Max Context length
        "max_seq_len_to_capture": <seq_len>, # Sequence length
        "quantization": "mxfp6", # Quantization
        "kv_cache_dtype": "mxint8", # KV cache compression
        "device": "qaic"
    }
    Copy to clipboard

Sample [config.pbtxt](https://github.com/triton-inference-server/vllm_backend/blob/r25.02/samples/model_repository/vllm_model/config.pbtxt)

Activate the vLLM virtual environment inside the Triton container before launching the Triton server:

source /opt/vllm-env/bin/activate
    Copy to clipboard

Set up Hugging Face credentials

huggingface-cli login <HF_TOKEN>
    Copy to clipboard

Configure number of cores according to NSP availability

export VLLM_QAIC_NUM_CORES=16
    Copy to clipboard

Start the Triton server

/opt/tritonserver/bin/tritonserver --model-repository=/opt/qti-aic/aic-triton-model-repositories/vllm_model
    Copy to clipboard

To use the `/completions` or `/chat/completions` endpoints, start OpenAI-compatible Triton server instead of running the binary above.

Prerequisite:

pip install /opt/tritonserver/python/tritonserver-*.whl
    cd /opt/tritonserver/python/openai && pip install -r requirements.txt
    Copy to clipboard

Start OpenAI-compatible Triton server

python3 openai_frontend/main.py --model-repository /opt/qti-aic/aic-triton-model-repositories/vllm_model/ --tokenizer <HF_MODEL_TAG>
    Copy to clipboard

The Triton server may take a few minutes (depending on the model) to download and compile the model.

Sample
[Client](https://github.com/triton-inference-server/vllm_backend/blob/r25.02/samples/client.py)
script is available in the sample model repository (built as part of
the Triton image using qaic-docker) `/opt/qti-aic/aic-triton-model-repositories/vllm_model/vllm_model`.

The sample client script (client.py) can be used to interface with
the Triton/vLLM inference server, and can be executed from the Triton
client environment.

User can also use the `generate` API to run inference from the Triton client
container:

curl -X POST localhost:8000/v2/models/vllm_model/generate -d '{"text_input": "My name is","parameters":{"stream":false, "temperature": 0, "max_tokens":1000}}'
    Copy to clipboard

See [Triton OpenAI User Guide](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html) for examples on using OpenAI endpoints for inferencing, benchmarking with genai-perf tool.

Last Published: Aug 25, 2026

[Previous Topic
Python backend for embeddings](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/python-embedding-backend.md) [Next Topic
Stable Diffusion](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/stable-diffusion.md)