# Vision-Language Model Inference on HTP

This tutorial shows how to build and deploy a **Large Multimodal Model (LMM)** —
a vision-language model that accepts both image and text inputs — on a Snapdragon
device using the Gen AI Builder API.

VL models such as **Qwen2.5-VL** and **Qwen3-VL** consist of two cooperating components:

- An **image encoder** (Vision Transformer) that converts pixel data into patch embeddings.
- A **text decoder** (causal LLM) that ingests the patch embeddings alongside the text
tokens and generates a response.

The builder handles each component separately and then wires them together using the
`WorkflowBuilder` API. For an overview of the single-model LLM builder, see
[LLM Inference on HTP](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder.html#genai-builder). For a full configuration reference, see
[Configuring the Gen AI Builder](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder_configuration.html#genai-builder-configuration).

Note

A runnable end-to-end example of the full LMM workflow is provided at
`examples/tutorials/multimodal_on_device_inference.py`.

## Configurations

- Host OS: Linux (x86\_64) with ADB (Android Debug Bridge) installed.
- Target Devices: Snapdragon Android Device
- Processor: Qualcomm NPU
- Backend: HTP

## Step 1: Setup

We recommend a machine with at least 32 GB of RAM. The vision encoder build is lighter
than a full LLM, but the text decoder still requires substantial memory.

### Prerequisites

This tutorial uses the `Qwen/Qwen2.5-VL-3B-Instruct` model as the primary example.
Qwen3-VL is also supported; see [Qwen3-VL](https://docs.qualcomm.com/doc/80-87189-2/topic/lmm_builder.html#lmm-qwen3vl) for differences.

- Download the Qwen2.5-VL-3B-Instruct model from Hugging Face:
[Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct).
- The guide assumes you have obtained the **quantized ONNX artifacts** for both the
vision encoder and language model components from the AIMET quantization notebooks.
These produce:

    - A vision encoder ONNX model with its `.encodings` file.
    - A language model ONNX model with its `.encodings` file.
    - Optionally: a pre-extracted embedding table (`.bin`) and its quantization
parameters (`.json`), if the embedding layer was split out during export.
- The guide uses a **Snapdragon SD 8 Elite (SM8750) Android device** to demonstrate
the workflow.

### Input Artifact Layout

VL model artifacts are split across two directories — one for the vision encoder
and one for the language model. A single `config.json` from the HuggingFace
repository serves both components.

<model_exports>/
        config.json                       # Shared HuggingFace model config
        tokenizer.json                    # Tokenizer (language model only)
        qwen_vit/
            onnx/
                <vision_model>.onnx       # Quantized vision encoder ONNX
                <vision_model>.encodings  # Quantization encodings (REQUIRED)
        qwen_llm/
            onnx/
                <language_model>.onnx     # Quantized language model ONNX
                <language_model>.encodings
        embedding/
            embedding_table.bin           # Pre-extracted embedding table (optional)
            embedding_quant_param.json    # Quant params for embedding table (optional)
    Copy to clipboard

Note

The `encodings_path` for the vision encoder **must** be provided. The Genie
runtime rejects float32 image-encoder inputs with:
`Unsupported input tensor pixel_values dtype QNN_DATATYPE_FLOAT_32`.
If you do not pass `encodings_path` explicitly, the builder searches for an
`.encodings` file alongside the ONNX; if none is found the build will succeed
but execution will fail at runtime.

import os
    from pathlib import Path
    
    import qairt
    from qairt import Device, DevicePlatformType
    from qairt.gen_ai_api.builders.gen_ai_builder_htp import EmbeddingInput, GenAIBuilderHTP
    from qairt.gen_ai_api.builders.workflow_builder import WorkflowBuilder
    from qairt.gen_ai_api.chat.chat_templates import HFChatTemplate
    from qairt.gen_ai_api.configs.workflow import WorkflowGraph, WorkflowNode, WorkflowNodeRole
    from qairt.gen_ai_api.containers.workflow_container import WorkflowContainer
    from qairt.gen_ai_api.executors.gen_ai_executable import GenerationRequest, TextGenerationResult
    from qairt.gen_ai_api.executors.image_t2t_executor import ImageT2TExecutor
    from qairt.gen_ai_api.gen_ai_builder_factory import GenAIBuilderFactory
    
    # Paths to model artifacts
    MODEL_EXPORTS     = "./qwen25vl_exports"
    VIT_ONNX_PATH     = f"{MODEL_EXPORTS}/qwen_vit/onnx/<vision_model>.onnx"
    VIT_ENCODINGS     = f"{MODEL_EXPORTS}/qwen_vit/onnx/<vision_model>.encodings"
    LLM_ONNX_PATH     = f"{MODEL_EXPORTS}/qwen_llm/onnx/<language_model>.onnx"
    CONFIG_JSON       = f"{MODEL_EXPORTS}/config.json"
    TOKENIZER_PATH    = f"{MODEL_EXPORTS}"
    EMBEDDING_BIN     = f"{MODEL_EXPORTS}/embedding/embedding_table.bin"
    EMBEDDING_PARAMS  = f"{MODEL_EXPORTS}/embedding/embedding_quant_param.json"
    Copy to clipboard

Tip

Set the environment variable **QAIRT\_TMP\_DIR** to an alternative temporary directory
with at least 30 GB free space to avoid disk pressure during the build.

export QAIRT_TMP_DIR=./vl_scratch
    Copy to clipboard

## Step 2: Build the Vision Encoder

Use `GenAIBuilderFactory.create_vision_encoder()` to create an architecture-aware
vision encoder builder. The factory reads `config.json` and automatically selects
the correct builder subclass (e.g. `Qwen2VLVisionEncoderBuilderHTP` for Qwen2.5-VL).

The `config_section` parameter tells the builder which subsection of `config.json`
holds the vision-specific architecture parameters (`hidden_size`, `num_heads`,
`patch_size`, RoPE theta, etc.). For Qwen2.5-VL and Qwen3-VL this is always
`"vision_config"`.

vision_builder = GenAIBuilderFactory.create_vision_encoder(
        pretrained_model_path=VIT_ONNX_PATH,
        cache_root="./vl_cache",
        config_path=CONFIG_JSON,
        config_section="vision_config",
        encodings_path=VIT_ENCODINGS,
    )
    
    vision_builder.set_targets(["chipset:SM8750"])
    Copy to clipboard

Note

The vision encoder build pipeline is simpler than the LLM pipeline. It does **not**
perform AR/CL conversion or model splitting. The stages are:
ONNX → MHA2SHA transform → Convert to DLC → Compile.
As a result, `set_transformation_options()` is not needed for the vision encoder.

## Step 3: Build the Text Decoder

Build the language model component using the same `GenAIBuilderFactory.create()`
factory as a standard LLM.

**Chat Template**

The chat template formats the user prompt (image + text) into the tokens the model
expects. Use `HFChatTemplate.from_pretrained()` with the HuggingFace model ID:

text_builder = GenAIBuilderFactory.create(
        Path(LLM_ONNX_PATH),
        "HTP",
        cache_root="./vl_cache",
        tokenizer_path=Path(TOKENIZER_PATH),
    )
    
    text_builder.chat_template = HFChatTemplate.from_pretrained(
        "Qwen/Qwen2.5-VL-3B-Instruct",
        kwargs={"cache_dir": "./vl_cache"},
    )
    
    text_builder.set_targets(["chipset:SM8750"])
    Copy to clipboard

**Embedding Table (when split out)**

If the quantization notebook extracted the embedding lookup table from the language
model ONNX, supply it via `EmbeddingInput` and disable the builder’s automatic split:

text_builder.embedding = EmbeddingInput(
        path=Path(EMBEDDING_BIN),
        quant_params_path=Path(EMBEDDING_PARAMS),
    )
    
    text_builder.set_transformation_options(options={
        "split.split_lm_head": False,
        "split.split_embedding": False,
    })
    Copy to clipboard

For details on why VL models require a pre-extracted embedding, the three `EmbeddingInput`
construction paths, and the quant params JSON format, see [Embedding Inputs](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder_configuration.html#genai-embedding-inputs).

Note

If your language model ONNX already contains the embedding layer (i.e., no
pre-extracted embedding table was produced during quantization), you can skip the
`EmbeddingInput` assignment and the `split_embedding` / `split_lm_head`
overrides.

## Step 4: Assemble the WorkflowBuilder

Describe the two-component topology with a `WorkflowGraph` and hand both builders to
`WorkflowBuilder.from_builders()`:

graph = WorkflowGraph(
        nodes=(
            WorkflowNode(
                name="vision",
                role=WorkflowNodeRole.IMAGE_ENCODER,
            ),
            WorkflowNode(
                name="text",
                role=WorkflowNodeRole.TEXT_GENERATOR,
                output_callback_type="textCallback",
            ),
        ),
        connections=(("vision", "text", "standard"),),
    )
    
    workflow_builder = WorkflowBuilder.from_builders(
        builders={"vision": vision_builder, "text": text_builder},
        workflow_graph=graph,
    )
    Copy to clipboard

Then trigger the build. Both components are built in the declared order, and the
resulting `WorkflowContainer` bundles both sub-containers together:

container = workflow_builder.build()
    container.save("./qwen25vl_container", exist_ok=True)
    Copy to clipboard

Tip

The build cache is keyed by a content hash of every option. Re-running with the same
configuration skips already-completed stages. Use `cache_root` to persist the cache
across runs. See [Understanding the Build Cache](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_overview.html#genai-builder-cache) for details.

## Step 5: Set up an Android device

Connect your Android device via ADB and set the `ANDROID_SERIAL` environment variable.

Obtain the ADB device ID by running:

adb devices
    Copy to clipboard

The output lists connected devices:

List of devices attached
    abcd1234   device
    Copy to clipboard

Set `ANDROID_SERIAL` to the device ID shown:

export ANDROID_SERIAL=abcd1234
    Copy to clipboard

If your device is connected to a remote machine, see the
[remote device troubleshooting](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder.html#remote-device-troubleshooting) section in
the [LLM Inference on HTP](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder.html#genai-builder) tutorial.

android_serial = os.getenv("ANDROID_SERIAL")
    android_hostname = os.getenv("ANDROID_HOSTNAME")
    
    device_id = f"{android_serial}@{android_hostname}" if android_hostname else android_serial
    android_device = Device(identifier=device_id, type=DevicePlatformType.ANDROID)
    Copy to clipboard

## Step 6: Generate Text from an Image

Load the saved container and obtain an executor. For workflows that contain an
`IMAGE_ENCODER` node the executor returned is an `ImageT2TExecutor`:

genai_container = WorkflowContainer.load(Path("./qwen25vl_container"))
    executor = genai_container.get_executor(android_device, clean_up=False)
    
    assert isinstance(executor, ImageT2TExecutor)
    Copy to clipboard

Construct a `GenerationRequest` with the image file and a text prompt. Pass a standard
image file (JPEG, PNG, WEBP) or a pre-processed `.raw` file — non-`.raw` files are
preprocessed automatically using the vision encoder’s processor:

request = GenerationRequest(
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "image", "image": "/path/to/image.jpg"},
                    {"type": "text", "text": "Describe this image."},
                ],
            }
        ]
    )
    
    result = executor.generate(request)
    
    if isinstance(result, TextGenerationResult):
        result.print()
    Copy to clipboard

The command above will print output similar to:

The image shows a mountainous landscape with snow-capped peaks ...
    
    Timing (microseconds):
    
    Init = 2034123 us
    Prompt Processing Time = 8241058 us
    Token Generation Rate = 5.43 toks/sec
    Copy to clipboard

Remove the on-device artifacts when done:

executor.clean_environment()
    Copy to clipboard

## Step 7 (Optional): Save and Load the WorkflowContainer

The `WorkflowContainer` can be saved to disk and reloaded later, either on the same
machine or after copying to a different host:

container.save("./qwen25vl_container", exist_ok=True)
    Copy to clipboard

from qairt.gen_ai_api.containers.workflow_container import WorkflowContainer
    
    loaded_container = WorkflowContainer.load(Path("./qwen25vl_container"))
    Copy to clipboard

## Qwen3-VL

Qwen3-VL models use the same API as Qwen2.5-VL. The factory automatically selects
`Qwen3VLVisionEncoderBuilderHTP` when it reads `Qwen3VLForConditionalGeneration`
from `config.json`, and selects `Qwen3VLTextBuilderHTP` for the text decoder.

No code changes are required — just substitute the Qwen3-VL artifact paths and the
HuggingFace model ID:

vision_builder = GenAIBuilderFactory.create_vision_encoder(
        pretrained_model_path="./qwen3vl_exports/qwen_vit/onnx/<vision_model>.onnx",
        cache_root="./vl_cache",
        config_path="./qwen3vl_exports/config.json",
        config_section="vision_config",
        encodings_path="./qwen3vl_exports/qwen_vit/onnx/<vision_model>.encodings",
    )
    
    text_builder = GenAIBuilderFactory.create(
        Path("./qwen3vl_exports/qwen_llm/onnx/<language_model>.onnx"),
        "HTP",
        cache_root="./vl_cache",
        tokenizer_path=Path("./qwen3vl_exports"),
    )
    text_builder.chat_template = HFChatTemplate.from_pretrained("Qwen/Qwen3-VL-4B-Instruct")
    
    # WorkflowGraph and WorkflowBuilder usage is identical to Qwen2.5-VL above.
    Copy to clipboard

Note

Qwen3-VL requires **wildcard pipeline connections** in the Genie runtime.
The factory sets this up automatically via
`VisionEncoderConfig.needs_wildcard_connection = True`; no user action is needed.

## Troubleshooting

### Build Errors

- **“No space left on device” or build runs out of disk space**
    - Set `QAIRT_TMP_DIR` to a volume with at least 30 GB free space.

export QAIRT_TMP_DIR=/path/to/large/volume/tmp
    Copy to clipboard

- **“Pretrained model path does not exist”**
    - Verify the path to the vision encoder ONNX and the language model ONNX both exist,
and that `config.json` is reachable via `config_path`.

- **Build runs out of memory**
    - The language model decoder typically needs 32–64 GB RAM. Increase swap space if
needed, and use `cache_root` to enable resumption.

### Runtime / Execution Errors

- **“Unsupported input tensor pixel\_values dtype QNN\_DATATYPE\_FLOAT\_32”**
    - The vision encoder was compiled without quantization encodings. Provide
`encodings_path` when calling `create_vision_encoder()`; the `.encodings`
file must correspond to the vision encoder ONNX.

- **“ValueError: This workflow contains an IMAGE\_ENCODER node and requires an image input”**
    - An image path was not supplied. Add `{"type": "image", "image": "/path/to/image.jpg"}`
(or a `.raw` file) to the `content` list of the user message.

- **“config\_section ‘vision\_config’ not found in config.json”**
    - The shared `config.json` does not contain a `"vision_config"` key. Confirm you
are using the top-level HuggingFace `config.json` for the VL model, not the
language-model-only config.

- **Response quality is poor or image content is ignored**
    - Confirm the image was loaded correctly. If passing a `.raw` file directly, verify it
was preprocessed using the same resize/normalize/channel-order pipeline that the
quantization notebook used when calibrating the vision encoder.

## Additional Tutorials

- [LLM Inference on HTP](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder.html) – Build and deploy a text-only LLM.
- [Low-Rank Adaptation (LoRA) Tutorial](https://docs.qualcomm.com/doc/80-87189-2/topic/lora_tutorial.html) – Deploy models with LoRA adapters.
- [Speculative Decoding Tutorial](https://docs.qualcomm.com/doc/80-87189-2/topic/speculative_decoding_tutorial.html) – Enable LADE, SSD, or Eaglet speculative decoding.

## Guides

- [Gen AI Builder Overview](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_overview.html#genai-overview) – Gen AI Builder architecture overview, including LMM support.
- [WorkflowBuilder Guide](https://docs.qualcomm.com/doc/80-87189-2/topic/workflow_builder.html#workflow-builder) – WorkflowBuilder API for composing multi-component workflows.
- [Configuring the Gen AI Builder](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder_configuration.html#genai-builder-configuration) – Full configuration reference for the builder API.
- [HTP Backend Extensions](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_backend_extensions.html#genai-backend-extensions) – HTP backend extensions JSON structure.

Last Published: Oct 23, 2026

[Previous Topic
Guides](https://docs.qualcomm.com/bundle/publicresource/80-87189-2/topics/genai_builder.md) [Next Topic
GGUF Inference on HTP](https://docs.qualcomm.com/bundle/publicresource/80-87189-2/topics/gguf_builder.md)