# Configuring the Gen AI Builder

This page documents all configuration options for the Gen AI Builder API. For a step-by-step
tutorial, see [LLM Inference on HTP](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder.html#genai-builder).

## Configuration Overview

The builder follows a layered configuration model. Only `create()` and `set_targets()` are
required – everything else has sensible defaults.

## Quick-Reference Table

| Method / Property | Default | Purpose |
| --- | --- | --- |
| `set_targets(["chipset:..."])` | *(required)* | Set target SoC for AOT compilation |
| `weight_sharing` | `True` | Share weights across AR/CL variants per split |
| `multi_graph` | `False` | Enable multiple context lengths [512, 1024, 2048, 3072, 4096] |
| `native_kv` | `False` | Enable native KV cache format (requires AR in {32, 64, 128, 256}) |
| `skip_ar_conversion` | `False` | Skip AR conversion; only vary context length |
| `encodings_path` | Auto-discovered | Override path to quantization encodings |
| `set_transformation_options()` | Auto-configured | Override model transformation settings |
| `set_compilation_options()` | From `set_targets()` | Override HTP compilation settings |
| `set_conversion_options()` | Auto-configured | Override ONNX-to-DLC conversion settings |
| `embedding` | `None` | Supply a pre-extracted embedding table ([Embedding Inputs](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder_configuration.html#genai-embedding-inputs)) |
| `embeddings` | Auto-configured | Unified `(source, mode)` view over the embedding LUT/model config |
| `lora_config` | `None` | Enable LoRA adapter support |
| `speculative_config` | `None` | Enable speculative decoding (LADE, SSD, or Eaglet) |
| `attach_model_for_arn()` | Not set | Pin a specific ONNX model to an AR value |

## Convenience Properties

These properties control the most common configuration choices.

### weight\_sharing

When `True` (default), the builder generates multiple AR variants (default `[1, 128]`)
and shares weights across them in each compiled split. This reduces binary size at the cost
of slightly more complex compilation.

builder.weight_sharing = True   # Default: ARN = [1, 128]
    Copy to clipboard

To disable weight sharing, first set AR to a single value (weight sharing requires multiple AR variants to share weights across):

builder.set_transformation_options(options={"arn": [128]})
    builder.weight_sharing = False
    Copy to clipboard

### multi\_graph

When `True`, the builder generates context binaries for multiple context lengths:
`[512, 1024, 2048, 3072, 4096]`. When `False` (default), only `[4096]` is used.

builder.multi_graph = True  # Context lengths: [512, 1024, 2048, 3072, 4096]
    Copy to clipboard

Note

For finer control over context lengths, use `set_transformation_options(options={"context_length": [...]})`
instead.

### native\_kv

When `True`, enables native KV cache format optimization. This also automatically sets
`permute_kv_cache_io=True` in the MHA2SHA transformation.

builder.set_transformation_options(options={"arn": [32, 128]})
    builder.native_kv = True
    Copy to clipboard

## Transformation Options

Transformation options control model structure (splitting, AR/CL variants, attention conversion).

### Usage

# Option A: Individual overrides via options dict (recommended)
    builder.set_transformation_options(options={
        "arn": [32, 128],
        "context_length": [2048, 4096, 6144, 8192],
        "split.num_splits": 4,
        "split.split_embedding": True,
    })
    
    # Option B: Full config object (advanced)
    from qairt.api.transforms.model_transformer_config import (
        SplitModelConfig,
        ModelTransformerConfig,
    )
    split_config = SplitModelConfig(split_embedding=True, num_splits=4)
    config = ModelTransformerConfig(split_model=split_config)
    builder.set_transformation_options(config=config)
    Copy to clipboard

### Transformation Keys and Defaults

The table below lists all supported `options` keys with their defaults. For full type
information and descriptions of each key, see
[`set_transformation_options()`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-gen-ai-api-builders-htp.html#qairt.gen_ai_api.builders.htp_mixin.HTPMixin.set_transformation_options).

| Key | Default |
| --- | --- |
| `arn` | `[1, 128]` (with weight\_sharing) |
| `context_length` | `[4096]` (or multi\_graph defaults) |
| `sliding_context_length` | `None` (only valid for SWA models; see [Sliding-window constraints](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder_configuration.html#genai-sliding-window-models)) |
| `split.num_splits` | Auto-calculated |
| `split.split_embedding` | `True` |
| `split.split_lm_head` | Varies by builder |
| `split.layer_boundary_name_regex` | `None` (residual-Add heuristic) |
| `mha2sha.permute_kv_cache_io` | `False` (auto when native\_kv) |
| `mha2sha.m2s_additional_start_points` | `[]` |
| `amoe.overridden_subselection` | Not set (MoE models only) |
| `amoe.remove_op_predicate` | Not set (MoE models only) |

#### How Split Count is Determined

When `split.num_splits` is not set explicitly, the builder auto-calculates it from the total
model parameter count. Because `split_embedding` and `split_lm_head` both default to `True`,
the embedding layer and LM head are each placed in their own split. The remaining decoder layers
are then divided into splits of approximately **2 GB** each:

num_splits = 3 + model_params // 2 GB
    Copy to clipboard

The base count of **3** accounts for:

1. **Embedding split** – the token-embedding layer
2. **LM-head split** – the language-model head layer
3. **First decoder split** – at least one split for decoder layers

Each additional 2 GB of parameters adds one more decoder split. For example, a 7B-parameter
model (roughly 7 GB at FP32) yields `3 + 7 // 2 = 6` splits.

#### Skipping Transformations

By default, the builder applies all relevant transformations (AR/CL conversion, model splitting,
MHA2SHA, and MoE adaptation for expert models). You can selectively disable individual
transformations for pre-transformed models, debugging, or custom workflows.

**Skip AR/CL Conversion**

Use the `skip_ar_conversion` property to keep the model at its original sequence length
without generating AR variants:

builder.skip_ar_conversion = True
    Copy to clipboard

This inserts a sentinel value `0` into the AR list, which is resolved at build time from the
model’s sequence length. Equivalently:

builder.set_transformation_options(options={"arn": [0]})
    Copy to clipboard

**Skip Model Splitting**

To produce a single unsplit model, set `num_splits=1` and disable embedding/LM-head extraction:

builder.set_transformation_options(options={
        "split.num_splits": 1,
        "split.split_embedding": False,
        "split.split_lm_head": False,
    })
    Copy to clipboard

**Skip MHA2SHA**

To disable the Multi-Head Attention to Single-Head Attention conversion, pass a full config with
`mha_config=None`. This requires the `config=` parameter since the options dict does not
support disabling an entire transformation:

from qairt.api.transforms.model_transformer_config import (
        ModelTransformerConfig,
        ARn_ContextLengthConfig,
        SplitModelConfig,
        MhaConfig,
    )
    
    builder.set_transformation_options(config=ModelTransformerConfig(
        arn_cl_options=ARn_ContextLengthConfig(),
        split_model=SplitModelConfig(num_splits=4, split_embedding=True, split_lm_head=True),
        mha_config=None,
    ))
    Copy to clipboard

**Skip MoE Adaptation**

MoE adaptation is only auto-enabled for models with an expert configuration (detected from
`config.json`). To disable it on an MoE model, pass a full config with `adapt_moe=None`:

builder.set_transformation_options(config=ModelTransformerConfig(
        arn_cl_options=ARn_ContextLengthConfig(),
        split_model=SplitModelConfig(num_splits=4, split_embedding=True, split_lm_head=True),
        mha_config=MhaConfig(),
        adapt_moe=None,
    ))
    Copy to clipboard

### Changing Sequence and Context Lengths

The `arn` (auto-regression number), `context_length`, and
`sliding_context_length` keys control the sequence/context variants the
builder produces. Source values are inferred from the exported graph; you
specify only the targets. Values combine Cartesian-style: the builder emits
`|AR| × |CL| × |SCL|` variants — each producing a graph named
`ar{ar}_cl{cl}[_scl{scl}][_{i}_of_{n}]`. Passing a `list[int]` for any key
adds a dimension to the product; a single value keeps that dimension at 1. The
`_scl<n>` suffix appears only on SWA models.

Note

Variants are packaged per split: for each `i`-th split (where the number
of splits is 1 or more), all variants for that split are combined into a
single weight-shared context binary.

Each snippet below can be used on its own; unset keys keep their defaults from
[Transformation Options](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder_configuration.html#genai-transformation-options).

**Changing sequence length (AR).**

builder.set_transformation_options(options={"arn": [1, 128]})
    Copy to clipboard

When `arn` is not set, most builders produce the default variants `[1, 128]`
(see [Transformation Options](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_builder_configuration.html#genai-transformation-options)) rather than preserving the source AR.
A few builder classes default to skipping AR conversion — check yours with
`print(builder.skip_ar_conversion)`, and set
`builder.skip_ar_conversion = True` explicitly to keep the source AR.

**Changing context length (CL).**

builder.set_transformation_options(options={"context_length": [4096, 8192]})
    Copy to clipboard

**Changing sliding context length (SCL, SWA models only).**

# For a model with sliding_window=1024 in its HF config.json
    builder.set_transformation_options(options={"sliding_context_length": [1280]})
    Copy to clipboard

**Graph naming example.** With `arn=[1, 128]`, `context_length=[4096]`,
`sliding_context_length=1280`, and `split.num_splits=3` on an SWA model,
the compiled graphs are named:

ar1_cl4096_scl1280_1_of_3
    ar1_cl4096_scl1280_2_of_3
    ar1_cl4096_scl1280_3_of_3
    ar128_cl4096_scl1280_1_of_3
    ar128_cl4096_scl1280_2_of_3
    ar128_cl4096_scl1280_3_of_3
    Copy to clipboard

**Sliding-window constraints.** For SWA models — those with a
`sliding_window` attribute in their HF `config.json` — the builder
additionally rewrites the SWA KV cache and attention mask via
[`change_sliding_context_length()`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-optimizer-passes-api.html#qairt.optimizer.onnx.change_sliding_context_length). When
`sliding_context_length` is not set, it defaults to
`ceil(sliding_window / 256) * 256`. User-supplied values are ceil-rounded to
a multiple of 256 (with a warning) and must satisfy
`sliding_window <= SCL <= min(context_length)` — otherwise `ValueError` is
raised. Setting `sliding_context_length` on a non-SWA model raises
`RuntimeError`.

## Embedding Inputs

The embedding lookup table is the first layer of a language model that maps integer token IDs to
dense float vectors. By default the builder expects this layer to be inside the ONNX model and
will split it out automatically (`split.split_embedding = True`). Some quantization workflows
(especially for VL models) extract this layer before handing the ONNX to the builder. In that
case you supply the pre-extracted table via `EmbeddingInput`.

### Three Paths

| Scenario | What to do | Why |
| --- | --- | --- |
| Embedding layer is inside the language model ONNX | Nothing; leave `split_embedding=True` (default) | Builder extracts, quantizes, and splits the layer automatically |
| Embedding layer was pre-extracted by a quantization notebook | Set `builder.embedding = EmbeddingInput(...)` and disable auto-split | Prevents the builder from trying to split a layer that is no longer there |
| Embedding layer **lives in its own ONNX file** | Set `builder.embeddings = (EmbeddingModel(model_path, encodings_path), EmbeddingMode.LUT)` | Extracts the table from the ONNX and attaches it in one step |

When you supply a pre-extracted table, disable the builder’s auto-split to avoid a double-split
error:

from qairt.gen_ai_api.builders.gen_ai_builder_htp import EmbeddingInput
    
    builder.embedding = EmbeddingInput(
        path=Path("./embedding/embedding_table.bin"),
        quant_params_path=Path("./embedding/embedding_quant_param.json"),
    )
    
    builder.set_transformation_options(options={
        "split.split_embedding": False,
        "split.split_lm_head": False,
    })
    Copy to clipboard

Note

`split_lm_head` should also be disabled when using a pre-extracted embedding if the
language model head was split out at the same time. Check the artifacts produced by your
quantization notebook. If a separate LM-head file was generated, disable both flags.

### Constructing EmbeddingInput

There are three ways to construct an `EmbeddingInput`, depending on what the quantization
pipeline produced:

**Direct construction** (most common, where AIMET notebook produced a quantized `.bin`):

embedding = EmbeddingInput(
        path=Path("embedding_table.bin"),
        quant_params_path=Path("embedding_quant_param.json"),  # optional
    )
    Copy to clipboard

When `quant_params_path` is `None`, the table is treated as unquantized float32.

**From a float32 binary + separate quant spec** (`EmbeddingInput.from_fp32_table`):

embedding = EmbeddingInput.from_fp32_table(
        fp32_lut_bin_path="./raw_embedding.bin",
        quant_params_path="./quant_spec.json",
        output_dir="./embedding_out",
    )
    Copy to clipboard

This quantizes the float table using the params in `quant_spec.json` and writes a
quantized `.bin` and companion parameters JSON into `output_dir`. Use this when the notebook gave
you a raw fp32 binary but you still have the quant spec.

**From an embedding-only ONNX** (`EmbeddingInput.from_onnx_model`):

embedding = EmbeddingInput.from_onnx_model(
        model_path="./embedding_model.onnx",
        encodings_path="./embedding_model.encodings",  # optional
        output_dir="./embedding_out",
    )
    Copy to clipboard

This extracts the embedding table from a standalone ONNX model (a Gather node consuming
`input_ids`). If `encodings_path` is provided the table is quantized during extraction;
otherwise the raw float table is saved.

**Via the builder’s unified property** (`embeddings`):

from qairt.gen_ai_api.builders.embedding import EmbeddingModel, EmbeddingMode
    
    builder.embeddings = (
        EmbeddingModel(
            model_path="./embedding_model.onnx",
            encodings_path="./embedding_model.encodings",  # optional
        ),
        EmbeddingMode.LUT,
    )
    Copy to clipboard

This is a convenience wrapper around `EmbeddingInput.from_onnx_model` that sets
`builder.embedding` in a single call, using the builder’s `cache_dir` as the output
directory.

### Quant Params JSON Format

The `quant_params_path` file describes a single per-tensor quantization. Two formats are
accepted:

**Internal format** (produced by the builder and `EmbeddingInput.from_fp32_table`):

{"bw": 8, "scale": [0.00123], "offset": [-132]}
    Copy to clipboard

**External / per-tensor format** (for example, from a separate quantization tool or AIMET):

[{"bitwidth": 8, "scale": 0.00123, "offset": -132, "dtype": "int"}]
    Copy to clipboard

Both are normalised to `bw` / `scale` / `offset` internally. Either format can be passed
to `EmbeddingInput(quant_params_path=...)`.

Fields:

| Field | Description |
| --- | --- |
| `bw` / `bitwidth` | Quantization bitwidth (4 or 8 are most common) |
| `scale` | Per-tensor scale factor (float). May be a scalar or a single-element list. |
| `offset` | Per-tensor zero-point offset (int). May be a scalar or a single-element list. |

### Why VL Models Need Pre-Extracted Embeddings

Vision-language models (Qwen2.5-VL, Qwen3-VL) share a single token-embedding lookup table
between the text pathway and the vision patch-embedding pathway. During quantization the
embedding layer is extracted from the language model ONNX so that both pathways can reference the
same weights at runtime via the Genie embedding mechanism. The resulting artifacts therefore
always include a separate `embedding_table.bin` and `embedding_quant_param.json`; the
language model ONNX no longer contains the embedding Gather node.

For a full build walkthrough that uses `EmbeddingInput`, see [Vision-Language Model Inference on HTP](https://docs.qualcomm.com/doc/80-87189-2/topic/lmm_builder.html#lmm-builder).

## Compilation Options

Compilation options control the HTP backend settings used during Ahead-of-Time compilation.

The builder exposes **two paths** for setting compilation options:

### Path A: Convenience Dict (Common Settings)

For the most commonly adjusted fields, use the `options` dict. This applies overrides
on top of the config created by `set_targets()`.

builder.set_compilation_options(options={
        "graphs.vtcm_size_in_mb": 8,
        "graphs.hvx_threads": 4,
        "graphs.optimization_type": 3,
        "devices.cores.perf_profile": "burst",
        "context.extended_udma": True,
    })
    Copy to clipboard

Important

`set_targets()` must be called **before** `set_compilation_options(options={...})`.
The options dict modifies the config that `set_targets()` creates.

The table below lists all supported convenience keys with their defaults. For full type
information and descriptions, see
[`set_compilation_options()`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-gen-ai-api-builders-htp.html#qairt.gen_ai_api.builders.htp_mixin.HTPMixin.set_compilation_options).

| Key | Default |
| --- | --- |
| `graphs.vtcm_size_in_mb` | `0` (device max) |
| `graphs.vtcm_size` | `0` (device max, in bytes) |
| `graphs.hvx_threads` | `0` (backend default) |
| `graphs.optimization_type` | `3` (from set\_targets) |
| `devices.cores.perf_profile` | `"burst"` (from set\_targets) |
| `context.extended_udma` | `False` |

### Path B: Full CompileConfig or Backend Extensions JSON

For settings **not covered** by the convenience dict (such as `fp16_relaxed_precision`,
`rpc_control_latency`, `pd_session`, `mem_type`, or `share_resources`), you need a full
`CompileConfig` object. There are two ways to obtain one:

1. **Load an existing backend extensions JSON file** using
`CompileConfig.from_backend_extensions()` or `populate_from_backend_extensions()`.
2. **Build one from Python** using the HTP config classes directly.

See [HTP Backend Extensions](https://docs.qualcomm.com/doc/80-87189-2/topic/genai_backend_extensions.html#genai-backend-extensions) for the JSON structure, Python construction examples,
the full list of HTP configuration classes, and round-trip serialization.

## Conversion Options

Conversion options control the ONNX-to-DLC conversion step. The builder auto-configures sensible
defaults (`act_precision=16`, `bias_precision=32`), so most builds do not need to call
`set_conversion_options()` at all.

Last Published: Sep 17, 2026

[Previous Topic
Replaying a stage from the cache](https://docs.qualcomm.com/bundle/publicresource/80-87189-2/topics/genai_overview.md) [Next Topic
HTP Backend Extensions](https://docs.qualcomm.com/bundle/publicresource/80-87189-2/topics/genai_backend_extensions.md)