# Vision-Language Model Inference on HTP

This tutorial shows how to build and deploy a **Large Multimodal Model (LMM)** —
a vision-language model that accepts both image and text inputs — on a Snapdragon
device using the Gen AI Builder API.

VL models such as **Qwen2.5-VL** and **Qwen3-VL** consist of two cooperating components:

- An **image encoder** (Vision Transformer) that converts pixel data into patch embeddings.
- A **text decoder** (causal LLM) that ingests the patch embeddings alongside the text
tokens and generates a response.

The builder handles each component separately and then wires them together using the
`WorkflowBuilder` API. For an overview of the single-model LLM builder, see
[LLM Inference on HTP](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder.html#genai-builder). For a full configuration reference, see
[Configuring the Gen AI Builder](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder_configuration.html#genai-builder-configuration).

Note

A runnable end-to-end example of the full LMM workflow is provided at
`examples/tutorials/multimodal_on_device_inference.py`.

## Configurations

- Host OS: Linux (x86\_64) with ADB (Android Debug Bridge) installed.
- Target Devices: Snapdragon Android Device
- Processor: Qualcomm NPU
- Backend: HTP

## Step 1: Setup

We recommend a machine with at least 32 GB of RAM. The vision encoder build is lighter
than a full LLM, but the text decoder still requires substantial memory.

### Prerequisites

This tutorial uses the `Qwen/Qwen2.5-VL-3B-Instruct` model as the primary example.
Qwen3-VL is also supported; see [Qwen3-VL](https://docs.qualcomm.com/doc/80-87189-2/topic/lmm_builder.html#lmm-qwen3vl) for differences.

- Download the Qwen2.5-VL-3B-Instruct model from Hugging Face:
[Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct).
- The guide assumes you have obtained the **quantized ONNX artifacts** for both the
vision encoder and language model components from the AIMET quantization notebooks.
These produce:

    - A vision encoder ONNX model with its `.encodings` file.
    - A language model ONNX model with its `.encodings` file.
    - Optionally: a pre-extracted embedding table (`.bin`) and its quantization
parameters (`.json`), if the embedding layer was split out during export.
- The guide uses a **Snapdragon SD 8 Elite (SM8750) Android device** to demonstrate
the workflow.

### Input Artifact Layout

VL model artifacts are split across two directories — one for the vision encoder
and one for the language model. A single `config.json` from the HuggingFace
repository serves both components.

<model_exports>/
        config.json                       # Shared HuggingFace model config
        tokenizer.json                    # Tokenizer (language model only)
        qwen_vit/
            onnx/
                <vision_model>.onnx       # Quantized vision encoder ONNX
                <vision_model>.encodings  # Quantization encodings (REQUIRED)
        qwen_llm/
            onnx/
                <language_model>.onnx     # Quantized language model ONNX
                <language_model>.encodings
        embedding/
            embedding_table.bin           # Pre-extracted embedding table (optional)
            embedding_quant_param.json    # Quant params for embedding table (optional)
    Copy to clipboard

Note

The `encodings_path` for the vision encoder **must** be provided. The Genie
runtime rejects float32 image-encoder inputs with:
`Unsupported input tensor pixel_values dtype QNN_DATATYPE_FLOAT_32`.
If you do not pass `encodings_path` explicitly, the builder searches for an
`.encodings` file alongside the ONNX; if none is found the build will succeed
but execution will fail at runtime.

import os
    from pathlib import Path
    
    import qairt
    from qairt import Device, DevicePlatformType
    from qairt.gen_ai_api.builders.gen_ai_builder_htp import EmbeddingInput, GenAIBuilderHTP
    from qairt.gen_ai_api.builders.workflow_builder import WorkflowBuilder
    from qairt.gen_ai_api.chat.chat_templates import HFChatTemplate
    from qairt.gen_ai_api.configs.workflow import WorkflowGraph, WorkflowNode, WorkflowNodeRole
    from qairt.gen_ai_api.containers.workflow_container import WorkflowContainer
    from qairt.gen_ai_api.executors.gen_ai_executable import GenerationRequest, TextGenerationResult
    from qairt.gen_ai_api.executors.image_t2t_executor import ImageT2TExecutor
    from qairt.gen_ai_api.gen_ai_builder_factory import GenAIBuilderFactory
    
    # Paths to model artifacts
    MODEL_EXPORTS     = "./qwen25vl_exports"
    VIT_ONNX_PATH     = f"{MODEL_EXPORTS}/qwen_vit/onnx/<vision_model>.onnx"
    VIT_ENCODINGS     = f"{MODEL_EXPORTS}/qwen_vit/onnx/<vision_model>.encodings"
    LLM_ONNX_PATH     = f"{MODEL_EXPORTS}/qwen_llm/onnx/<language_model>.onnx"
    CONFIG_JSON       = f"{MODEL_EXPORTS}/config.json"
    TOKENIZER_PATH    = f"{MODEL_EXPORTS}"
    EMBEDDING_BIN     = f"{MODEL_EXPORTS}/embedding/embedding_table.bin"
    EMBEDDING_PARAMS  = f"{MODEL_EXPORTS}/embedding/embedding_quant_param.json"
    Copy to clipboard

Tip

Set the environment variable **QAIRT\_TMP\_DIR** to an alternative temporary directory
with at least 30 GB free space to avoid disk pressure during the build.

export QAIRT_TMP_DIR=./vl_scratch
    Copy to clipboard

## Step 2: Build the Vision Encoder

Use `GenAIBuilderFactory.create_vision_encoder()` to create an architecture-aware
vision encoder builder. The factory reads `config.json` and automatically selects
the correct builder subclass (e.g. `Qwen2VLVisionEncoderBuilderHTP` for Qwen2.5-VL).

The `config_section` parameter tells the builder which subsection of `config.json`
holds the vision-specific architecture parameters (`hidden_size`, `num_heads`,
`patch_size`, RoPE theta, etc.). For Qwen2.5-VL and Qwen3-VL this is always
`"vision_config"`.

vision_builder = GenAIBuilderFactory.create_vision_encoder(
        pretrained_model_path=VIT_ONNX_PATH,
        cache_root="./vl_cache",
        config_path=CONFIG_JSON,
        config_section="vision_config",
        encodings_path=VIT_ENCODINGS,
    )
    
    vision_builder.set_targets(["chipset:SM8750"])
    Copy to clipboard

Note

The vision encoder build pipeline is simpler than the LLM pipeline. It does **not**
perform AR/CL conversion or model splitting. The stages are:
ONNX → MHA2SHA transform → Convert to DLC → Compile.
As a result, `set_transformation_options()` is not needed for the vision encoder.

## Step 3: Build the Text Decoder

Build the language model component using the same `GenAIBuilderFactory.create()`
factory as a standard LLM.

**Chat Template**

The chat template formats the user prompt (image + text) into the tokens the model
expects. Use `HFChatTemplate.from_pretrained()` with the HuggingFace model ID:

text_builder = GenAIBuilderFactory.create(
        Path(LLM_ONNX_PATH),
        "HTP",
        cache_root="./vl_cache",
        tokenizer_path=Path(TOKENIZER_PATH),
    )
    
    text_builder.chat_template = HFChatTemplate.from_pretrained(
        "Qwen/Qwen2.5-VL-3B-Instruct",
        kwargs={"cache_dir": "./vl_cache"},
    )
    
    text_builder.set_targets(["chipset:SM8750"])
    Copy to clipboard

**Embedding Table (when split out)**

If the quantization notebook extracted the embedding lookup table from the language
model ONNX, supply it via `EmbeddingInput` and disable the builder’s automatic split:

text_builder.embedding = EmbeddingInput(
        path=Path(EMBEDDING_BIN),
        quant_params_path=Path(EMBEDDING_PARAMS),
    )
    
    text_builder.set_transformation_options(options={
        "split.split_lm_head": False,
        "split.split_embedding": False,
    })
    Copy to clipboard

For details on why VL models require a pre-extracted embedding, the three `EmbeddingInput`
construction paths, and the quant params JSON format, see [Embedding Inputs](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder_configuration.html#genai-embedding-inputs).

Note

If your language model ONNX already contains the embedding layer (i.e., no
pre-extracted embedding table was produced during quantization), you can skip the
`EmbeddingInput` assignment and the `split_embedding` / `split_lm_head`
overrides.

## Step 4: Assemble the WorkflowBuilder

Describe the two-component topology with a `WorkflowGraph` and hand both builders to
`WorkflowBuilder.from_builders()`:

graph = WorkflowGraph(
        nodes=(
            WorkflowNode(
                name="vision",
                role=WorkflowNodeRole.IMAGE_ENCODER,
            ),
            WorkflowNode(
                name="text",
                role=WorkflowNodeRole.TEXT_GENERATOR,
                output_callback_type="textCallback",
            ),
        ),
        connections=(("vision", "text", "standard"),),
    )
    
    workflow_builder = WorkflowBuilder.from_builders(
        builders={"vision": vision_builder, "text": text_builder},
        workflow_graph=graph,
    )
    Copy to clipboard

Then trigger the build. Both components are built in the declared order, and the
resulting `WorkflowContainer` bundles both sub-containers together:

container = workflow_builder.build()
    container.save("./qwen25vl_container", exist_ok=True)
    Copy to clipboard

Tip

The build cache is keyed by a content hash of every option. Re-running with the same
configuration skips already-completed stages. Use `cache_root` to persist the cache
across runs. See [Understanding the Build Cache](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_overview.html#genai-builder-cache) for details.

## Step 5: Set up an Android device

Connect your Android device via ADB and set the `ANDROID_SERIAL` environment variable.

Obtain the ADB device ID by running:

adb devices
    Copy to clipboard

The output lists connected devices:

List of devices attached
    abcd1234   device
    Copy to clipboard

Set `ANDROID_SERIAL` to the device ID shown:

export ANDROID_SERIAL=abcd1234
    Copy to clipboard

If your device is connected to a remote machine, see the
[remote device troubleshooting](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder.html#remote-device-troubleshooting) section in
the [LLM Inference on HTP](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder.html#genai-builder) tutorial.

android_serial = os.getenv("ANDROID_SERIAL")
    android_hostname = os.getenv("ANDROID_HOSTNAME")
    
    device_id = f"{android_serial}@{android_hostname}" if android_hostname else android_serial
    android_device = Device(identifier=device_id, type=DevicePlatformType.ANDROID)
    Copy to clipboard

## Step 6: Generate Text from an Image

Load the saved container and obtain an executor. For workflows that contain an
`IMAGE_ENCODER` node the executor returned is an `ImageT2TExecutor`:

genai_container = WorkflowContainer.load(Path("./qwen25vl_container"))
    executor = genai_container.get_executor(android_device, clean_up=False)
    
    assert isinstance(executor, ImageT2TExecutor)
    Copy to clipboard

Construct a `GenerationRequest` with the image file and a text prompt. The image must
be a pre-processed `.raw` file (pixel data in the format expected by the vision
encoder — typically `uint8` RGB or the format produced by the preparation notebook):

request = GenerationRequest(
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "image", "image": "/path/to/image.raw"},
                    {"type": "text", "text": "Describe this image."},
                ],
            }
        ]
    )
    
    result = executor.generate(request)
    
    if isinstance(result, TextGenerationResult):
        result.print()
    Copy to clipboard

The command above will print output similar to:

The image shows a mountainous landscape with snow-capped peaks ...
    
    Timing (microseconds):
    
    Init = 2034123 us
    Prompt Processing Time = 8241058 us
    Token Generation Rate = 5.43 toks/sec
    Copy to clipboard

Remove the on-device artifacts when done:

executor.clean_environment()
    Copy to clipboard

## Step 7 (Optional): Save and Load the WorkflowContainer

The `WorkflowContainer` can be saved to disk and reloaded later, either on the same
machine or after copying to a different host:

container.save("./qwen25vl_container", exist_ok=True)
    Copy to clipboard

from qairt.gen_ai_api.containers.workflow_container import WorkflowContainer
    
    loaded_container = WorkflowContainer.load(Path("./qwen25vl_container"))
    Copy to clipboard

## Qwen3-VL

Qwen3-VL models use the same API as Qwen2.5-VL. The factory automatically selects
`Qwen3VLVisionEncoderBuilderHTP` when it reads `Qwen3VLForConditionalGeneration`
from `config.json`, and selects `Qwen3VLTextBuilderHTP` for the text decoder.

No code changes are required — just substitute the Qwen3-VL artifact paths and the
HuggingFace model ID:

vision_builder = GenAIBuilderFactory.create_vision_encoder(
        pretrained_model_path="./qwen3vl_exports/qwen_vit/onnx/<vision_model>.onnx",
        cache_root="./vl_cache",
        config_path="./qwen3vl_exports/config.json",
        config_section="vision_config",
        encodings_path="./qwen3vl_exports/qwen_vit/onnx/<vision_model>.encodings",
    )
    
    text_builder = GenAIBuilderFactory.create(
        Path("./qwen3vl_exports/qwen_llm/onnx/<language_model>.onnx"),
        "HTP",
        cache_root="./vl_cache",
        tokenizer_path=Path("./qwen3vl_exports"),
    )
    text_builder.chat_template = HFChatTemplate.from_pretrained("Qwen/Qwen3-VL-4B-Instruct")
    
    # WorkflowGraph and WorkflowBuilder usage is identical to Qwen2.5-VL above.
    Copy to clipboard

Note

Qwen3-VL requires **wildcard pipeline connections** in the Genie runtime.
The factory sets this up automatically via
`VisionEncoderConfig.needs_wildcard_connection = True`; no user action is needed.

## Troubleshooting

### Build Errors

- **“No space left on device” or build runs out of disk space**
    - Set `QAIRT_TMP_DIR` to a volume with at least 30 GB free space.

export QAIRT_TMP_DIR=/path/to/large/volume/tmp
    Copy to clipboard

- **“Pretrained model path does not exist”**
    - Verify the path to the vision encoder ONNX and the language model ONNX both exist,
and that `config.json` is reachable via `config_path`.

- **Build runs out of memory**
    - The language model decoder typically needs 32–64 GB RAM. Increase swap space if
needed, and use `cache_root` to enable resumption.

### Runtime / Execution Errors

- **“Unsupported input tensor pixel\_values dtype QNN\_DATATYPE\_FLOAT\_32”**
    - The vision encoder was compiled without quantization encodings. Provide
`encodings_path` when calling `create_vision_encoder()`; the `.encodings`
file must correspond to the vision encoder ONNX.

- **“ValueError: This workflow contains an IMAGE\_ENCODER node and requires an image input”**
    - An image path was not supplied. Pass `--image /path/to/image.raw` (script) or
set `{"type": "image", "image": "..."}` in the `GenerationRequest` content.

- **“config\_section ‘vision\_config’ not found in config.json”**
    - The shared `config.json` does not contain a `"vision_config"` key. Confirm you
are using the top-level HuggingFace `config.json` for the VL model, not the
language-model-only config.

- **Response quality is poor or image content is ignored**
    - Verify that the `.raw` image file was preprocessed using the same pipeline (resize,
normalize, channel order) that the quantization notebook used when calibrating the
vision encoder. Mismatched preprocessing is the most common cause of poor VL output.

## Additional Tutorials

- [LLM Inference on HTP](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder.html) – Build and deploy a text-only LLM.
- [Low-Rank Adaptation (LoRA) Tutorial](https://docs.qualcomm.com/doc/80-87189-2/topic/lora_tutorial.html) – Deploy models with LoRA adapters.
- [Speculative Decoding Tutorial](https://docs.qualcomm.com/doc/80-87189-2/topic/speculative_decoding_tutorial.html) – Enable LADE, SSD, or Eaglet speculative decoding.

## Guides

- [Gen AI Builder Overview](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_overview.html#genai-overview) – Gen AI Builder architecture overview, including LMM support.
- [WorkflowBuilder Guide](https://docs.qualcomm.com/doc/80-87189-2/topic/workflow_builder.html#workflow-builder) – WorkflowBuilder API for composing multi-component workflows.
- [Configuring the Gen AI Builder](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder_configuration.html#genai-builder-configuration) – Full configuration reference for the builder API.
- [HTP Backend Extensions](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_backend_extensions.html#genai-backend-extensions) – HTP backend extensions JSON structure.

Last Published: Aug 26, 2026

[Previous Topic
Guides](https://docs.qualcomm.com/bundle/publicresource/80-87189-2/topics/genai_builder.md) [Next Topic
GGUF Inference on HTP](https://docs.qualcomm.com/bundle/publicresource/80-87189-2/topics/gguf_builder.md)