# LLM Pipeline Stages

Built-in stages for the LLM pipeline. Each stage implements the three-class
contract (Input, Config, Output) and is registered in the global stage registry.

## ModelLoadingStage

Model Loading Stage

This stage handles loading and re-authoring models using QcAutoModelForCausalLM.

- *class* qairt.experimental.pipeline.torch.llm.stages.model\_loading\_stage.ExportFormat(*value*)

    - Bases: `str`, `Enum`

Supported export formats for `ModelLoadingStage.export()`.

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.model\_loading\_stage.ModelLoadingConfig

    - Bases: [`StageConfig`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageConfig)

Configuration for model loading stage.

- hf\_config

    - Optional HuggingFace config to pass to QcAutoModelForCausalLM.

- model\_config\_overrides

    - Optional overrides for model config (HF config + QC config)
after loading.

- hf\_pretrained\_kwargs

    - kwargs forwarded to QcAutoModelForCausalLM.from\_pretrained
(e.g. trust\_remote\_code, revision, etc.).

- model\_reauthoring

    - Whether to re-author the model via QcAutoModelForCausalLM.

- qc\_config

    - Populated by \_pre\_hook when model\_reauthoring=True.

- load\_tokenizer

    - Whether to load the tokenizer.

- hf\_tokenizer\_kwargs

    - Forwarded to AutoTokenizer.from\_pretrained.

- apply\_default\_adaptations

    - Whether to apply default backend adaptations.

- custom\_reauthoring\_artifacts

    - Directory with mappings.py + reauthoring.py; requires model\_reauthoring=True.

- Validators

    - - `_check_custom_reauthoring_requires_reauthoring` » `all fields`

- *field* apply\_default\_adaptations*: bool*  *= True*

    - - Validated by

    - - `_check_custom_reauthoring_requires_reauthoring`

- *field* custom\_reauthoring\_artifacts*: Optional[Path]*  *= None*

    - - Validated by

    - - `_check_custom_reauthoring_requires_reauthoring`

- *field* hf\_config*: PretrainedConfig | None*  *= None*

    - - Validated by

    - - `_check_custom_reauthoring_requires_reauthoring`

- *field* hf\_pretrained\_kwargs*: Dict[str, Any]*  *[Optional]*

    - - Validated by

    - - `_check_custom_reauthoring_requires_reauthoring`

- *field* hf\_tokenizer\_kwargs*: Dict[str, Any]*  *[Optional]*

    - - Validated by

    - - `_check_custom_reauthoring_requires_reauthoring`

- *field* load\_tokenizer*: bool*  *= True*

    - - Validated by

    - - `_check_custom_reauthoring_requires_reauthoring`

- *field* model\_config\_overrides*: Dict[str, Any]*  *[Optional]*

    - - Validated by

    - - `_check_custom_reauthoring_requires_reauthoring`

- *field* model\_id\_or\_path*: Optional[str]*  *= None*

    - - Validated by

    - - `_check_custom_reauthoring_requires_reauthoring`

- *field* model\_reauthoring*: bool*  *= True*

    - - Validated by

    - - `_check_custom_reauthoring_requires_reauthoring`

- *field* qc\_config*: Optional[Any]*  *= None*

    - - Validated by

    - - `_check_custom_reauthoring_requires_reauthoring`

- model\_post\_init(*context: Any*, */*) → None

    - We need to both initialize private attributes and call the user-defined model\_post\_init
method.

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.model\_loading\_stage.ModelLoadingInput

    - Bases: [`StageInput`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageInput)

Optional upstream inputs for model loading.

All fields are optional so the stage can start a pipeline (no upstream dependencies).
When a preceding stage provides a model in memory or exports an HF directory, these
fields are auto-wired from the predecessor’s output by field name.

- *field* config*: Optional[Any]*  *= None*

    - 

- *field* model*: Optional[torch.nn.Module]*  *= None*

    - 

- *field* model\_path*: Optional[str]*  *= None*

    - 

- *field* tokenizer*: Optional[Any]*  *= None*

    -

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.model\_loading\_stage.ModelLoadingOutput

    - Bases: [`StageOutput`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageOutput)

Output from model loading stage.

- *field* config*: Optional[Any]*  *= None*

    - 

- *field* fp\_model*: Optional[torch.nn.Module]*  *= None*

    - 

- *field* is\_reauthored*: bool*  *= False*

    - 

- *field* lora\_metadata*: Optional['LoRAAdapterMetadata']*  *= None*

    - 

- *field* model*: torch.nn.Module*  *[Required]*

    - 

- *field* tokenizer*: Any | None*  *= None*

    -

- *class* qairt.experimental.pipeline.torch.llm.stages.model\_loading\_stage.ModelLoadingStage

    - Bases: [`Stage`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.Stage)

Stage for loading and re-authoring models.

- Config

    - alias of `ModelLoadingConfig`

- Input

    - alias of `ModelLoadingInput`

- Output

    - alias of `ModelLoadingOutput`

- evaluate(*stage\_output: ModelLoadingOutput*, *config: ModelLoadingConfig*, *\*\*kwargs: Any*) → Dict[str, float]

    - Run the configured evaluation metrics on the loaded model.

- Parameters

    - - **stage\_output** – Output from this stage, providing the model and tokenizer.
- **config** – Stage configuration carrying `evaluator_config` and, via the
pipeline context, the `cache_dir` default output location.
- **\*\*kwargs** – Only `evaluator_config` is accepted, to override the one on the
stage/pipeline config; it takes precedence when provided. Any other
keyword argument raises `TypeError` (guards against silent typos).

- Returns

    - The `{display_name: score}` mapping from `run_evaluation()`.

- Raises

    - - **ValueError** – If no `evaluator_config` with metrics is configured, if the
    tokenizer was not loaded, or if a reauthored model is evaluated without
    a `generator_config` (required for sequence/context lengths).
- **TypeError** – If an unexpected keyword argument is passed.

- *classmethod* export(*stage\_output: ModelLoadingOutput*, *output\_path: Path*, *format: ExportFormat = ExportFormat.HF*, *\*\*kwargs: Any*) → None

    - Export the loaded model in the requested format.

- Parameters

    - - **stage\_output** – Output containing the loaded model.
- **output\_path** – Destination directory (HF) or file path (ONNX).
- **format** – `HF` (default) or `ONNX`.
- **\*\*kwargs** – Forwarded to the underlying export function.

- generate(*stage\_output: ModelLoadingOutput*, *config: ModelLoadingConfig*, *prompt: Union[str, list[dict[str, str]], Path]*, *\*\*kwargs: Any*) → Any

    - Optional generation method using the loaded model.

- Parameters

    - - **stage\_output** – Output from the model loading stage.
- **config** – Stage configuration.
- **prompt** – Input text to generate from. Can be a string, a Path to a text file,
or a list of chat messages (in which case the last message content is used).
- **\*\*kwargs** – Additional keyword arguments forwarded to the underlying generate.

- *classmethod* load\_from\_cache(*cache\_path: Path*, *\*\*kwargs: Any*) → ModelLoadingOutput

    - Restore a previously exported model from `cache_path`.

Loads the fully adapted model (including ConvInplaceLinear) directly from
the torch-saved artifact, then re-applies the KV cache class-level patches
for re-authored models (these are not preserved in pickle).

- Keyword Arguments

    - **lora\_cfg** – Optional LoRAFeatureConfig to recompute adapter metadata
(not persisted to cache).

## QuantizationStage

Quantization Stage

Runs a named AIMET quantization recipe against a loaded model and produces
a quantized model artifact (QuantizationSimModel + encodings).

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.quantization\_stage.QuantizationConfig

    - Bases: [`StageConfig`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageConfig)

Configuration for the quantization stage.

- recipe\_name

    - Registry key for the recipe to run (e.g. `"lpbq_seqmse"`).

- recipe\_config

    - Recipe configuration — either a path to a YAML file or an
inline dict.  `None` falls back to the recipe class default config.

- technique\_kwargs

    - Per-technique parameter overrides forwarded verbatim as
top-level config overrides.  Use nested dicts keyed by technique name,
e.g. `{"seqmse": {"num_batches": 10}, "lpbq": {"decompressed_bw": 4}}`.

- config\_overrides

    - Arbitrary top-level config overrides applied on top of
`recipe_config` before technique\_kwargs.

- filename\_prefix

    - Prefix for exported ONNX/encodings filenames
(e.g. `"Llama-3.2-1B-Instruct_base"`). Defaults to `"model"`.

- export\_format

    - Controls which export artifacts are written.
`"v2"` (default) writes only the `sim.onnx.export()` artifacts.
`"v1"` writes only the deprecated `quantsim.export()` artifacts.
`"all"` writes both.

- generator

    - Optional pre-built `LLMGenerator` for static-shape forward passes.
Level-3 Python API users can pass a generator instance directly here
instead of letting the recipe build one internally.

- post\_prepare\_evaluator\_config

    - Optional config to evaluate the prepared
model right after `prepare_llm` and before quantsim creation. Runs
only when `enabled`; results are written to `model_preparation_path`.
Independent of `evaluator_config` (which drives on-demand `evaluate()`).
Excluded from the stage cache key since it never affects the quantized artifact.

- dsp\_arch

    - DSP architecture selecting the HTP quantsim config (e.g. `"v81"`).
Filled from `soc_details` when unset so it participates in the cache key.
A per-technique `dsp_arch` in the recipe config takes precedence.

- *field* attention\_mask\_min*: int*  *= -100*

    - 

- *field* bypass\_adapted\_forward*: bool*  *= False*

    - 

- *field* config\_overrides*: Dict[str, Any]*  *[Optional]*

    - 

- *field* dsp\_arch*: Optional[str]*  *= None*

    - 

- *field* eaglet\_config*: Optional[Dict[str, Any]]*  *= None*

    - 

- *field* export\_format*: Literal['v1', 'v2', 'all']*  *= 'v2'*

    - 

- *field* filename\_prefix*: Optional[str]*  *= None*

    - 

- *field* lora*: Optional[[LoRAFeatureConfig](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-lora.html#qairt.experimental.pipeline.torch.llm.lora.configs.LoRAFeatureConfig)]*  *= None*

    - 

- *field* model\_preparation\_path*: Optional[str]*  *= None*

    - 

- *field* post\_prepare\_evaluator\_config*: Optional[[PostPrepareEvaluatorConfig](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-common.html#qairt.experimental.pipeline.torch.common.configs.PostPrepareEvaluatorConfig)]*  *= None*

    - 

- *field* recipe\_config*: Optional[Union[str, Dict[str, Any]]]*  *= None*

    - 

- *field* recipe\_name*: str*  *[Required]*

    - 

- *field* technique\_kwargs*: Dict[str, Any]*  *[Optional]*

    - 

- *field* test\_vector\_config*: Optional[[TestVectorConfig](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-common.html#qairt.experimental.pipeline.torch.common.configs.TestVectorConfig)]*  *= None*

    - 

- model\_post\_init(*context: Any*, */*) → None

    - We need to both initialize private attributes and call the user-defined model\_post\_init
method.

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.quantization\_stage.QuantizationInput

    - Bases: [`StageInput`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageInput)

Upstream model and tokenizer provided by the model loading stage.

- *field* config*: Optional[Any]*  *= None*

    - 

- *field* lora\_metadata*: LoRAAdapterMetadata | None*  *= None*

    - 

- *field* model*: torch.nn.Module*  *[Required]*

    - 

- *field* tokenizer*: Any | None*  *= None*

    -

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.quantization\_stage.QuantizationOutput

    - Bases: [`StageOutput`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageOutput)

Output from the quantization stage.

- model

    - The reauthored model wrapping `quantsim.model`.

- quantsim

    - AIMET QuantizationSimModel; `None` for optimization-only recipes
or when loading from cache.

- quantizer

    - The `AIMETQuantizer` instance used during quantization.  Retained
so that `export()` can delegate to `AIMETQuantizer.export()`.
`None` when loading from cache.

- tokenizer

    - Passed through from upstream stage.

- config

    - HuggingFace model config passed through from upstream stage.

- generator

    - Optional LLMGenerator wrapping the quantized model for evaluation.
Set when the model was prepared via `prepare_llm` so downstream evaluation
receives a ready-to-use model without signature inspection.

- export\_format

    - The export format used when writing artifacts — `"v2"`, `"v1"`,
or `"all"`.  Populated by `export()` and `load_from_cache()` so
downstream stages know which layout was written.

- model\_path

    - Path to exported ONNX model; populated after export or cache restore.

- encodings\_path

    - Path to exported encodings file; populated after export or cache restore.

- tokenizer\_path

    - Path to tokenizer directory; populated after export or cache restore.

- config\_path

    - Path to exported config.json; populated after export or cache restore.

- base\_torch\_encodings\_path

    - Path to the base model’s torch-format encodings JSON;
populated when LoRA is configured.

- base\_node\_mapping\_path

    - Path to the ONNX node mapping JSON for LoRA attach points;
populated when LoRA is configured.

- lora\_metadata\_path

    - Path to the top-level LoRA metadata YAML; populated when LoRA
is configured.

- lora\_metadata

    - LoRA adapter metadata passed through from upstream.

- fp\_model

    - The untouched floating-point model passed into quantization. Forwarded
so downstream stages (e.g. draft-model loading) can build from the original
HF-style model, while `model` holds the quantization product itself.

- *field* base\_node\_mapping\_path*: Optional[str]*  *= None*

    - 

- *field* base\_torch\_encodings\_path*: Optional[str]*  *= None*

    - 

- *field* config*: Optional[Any]*  *= None*

    - 

- *field* config\_path*: Optional[str]*  *= None*

    - 

- *field* encodings\_path*: Optional[str]*  *= None*

    - 

- *field* export\_format*: Literal['v1', 'v2', 'all']*  *= 'v2'*

    - 

- *field* filename\_prefix*: Optional[str]*  *= None*

    - 

- *field* fp\_model*: Optional[torch.nn.Module]*  *= None*

    - 

- *field* generator*: Optional[Any]*  *= None*

    - 

- *field* hidden\_states\_cache\_path*: Optional[str]*  *= None*

    - 

- *field* lora\_builder\_config*: Optional[[LoraBuilderInputConfig](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-gen-ai-modules-lora.html#qairt.modules.lora.lora_config.LoraBuilderInputConfig)]*  *= None*

    - 

- *field* lora\_config*: [LoRAFeatureConfig](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-lora.html#qairt.experimental.pipeline.torch.llm.lora.configs.LoRAFeatureConfig) | None*  *= None*

    - 

- *field* lora\_metadata*: LoRAAdapterMetadata | None*  *= None*

    - 

- *field* lora\_metadata\_path*: Optional[str]*  *= None*

    - 

- *field* lora\_use\_case\_artifacts*: Optional[Dict[str, Any]]*  *= None*

    - 

- *field* model*: Optional[torch.nn.Module]*  *= None*

    - 

- *field* model\_path*: Optional[str]*  *= None*

    - 

- *field* quantizer*: Optional[AIMETQuantizer]*  *= None*

    - 

- *field* quantsim*: Optional[Any]*  *= None*

    - 

- *field* test\_vectors\_path*: Optional[str]*  *= None*

    - 

- *field* tokenizer*: Any | None*  *= None*

    - 

- *field* tokenizer\_path*: Optional[str]*  *= None*

    -

- *class* qairt.experimental.pipeline.torch.llm.stages.quantization\_stage.QuantizationStage

    - Bases: [`Stage`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.Stage)[`QuantizationInput`, `QuantizationConfig`, `QuantizationOutput`]

Runs a named AIMET quantization recipe and produces a quantized model artifact.

- Config

    - alias of `QuantizationConfig`

- Input

    - alias of `QuantizationInput`

- Output

    - alias of `QuantizationOutput`

- evaluate(*stage\_output: QuantizationOutput*, *config: QuantizationConfig*, *\*\*kwargs: Any*) → dict[str, float]

    - Run the configured evaluation metrics on the quantized model.

- Parameters

    - - **stage\_output** – Output from the quantization stage (model, tokenizer, config,
and optionally a pre-wrapped generator).
- **config** – Stage configuration carrying `evaluator_config` and, via the
pipeline context, the `cache_dir` default output location.
- **\*\*kwargs** – Only `evaluator_config` is accepted, to override the one on the
stage/pipeline config; it takes precedence when provided. Any other
keyword argument raises `TypeError` (guards against silent typos).

- Returns

    - The `{display_name: score}` mapping from `run_evaluation()`.

- Raises

    - - **ValueError** – If no `evaluator_config` with metrics is configured, the
    tokenizer is missing, or sequence/context lengths are unavailable for
    wrapping the model.
- **TypeError** – If an unexpected keyword argument is passed.

- *classmethod* export(*stage\_output: QuantizationOutput*, *output\_path: Path*, *\*\*kwargs: Any*) → None

    - Export the quantized model to `output_path` (ONNX + encodings + tokenizer + config).

Delegates to `QuantizationResult.export()` which handles ONNX export,
tokenizer, and model config saving in one call.

Populates `model_path`, `encodings_path`, `tokenizer_path`, and
`config_path` on `stage_output` so downstream stages can consume them.

When `lora_config` and `lora_metadata` are provided, additionally exports
base torch encodings, an ONNX node mapping, and a top-level LoRA metadata
YAML file, populating the corresponding output fields.

- Parameters

    - - **stage\_output** – Output from this stage.
- **output\_path** – Destination directory.
- **\*\*kwargs** – `filename_prefix` (default `"model"`).
`export_format`: override the export format; defaults to
`stage_output.export_format` (`"v2"`, `"v1"`, or `"all"`).
When `"all"`, both v2 ONNX artifacts and v1 encodings are written.

- generate(*stage\_output: QuantizationOutput*, *config: QuantizationConfig*, *prompt: Union[str, list[dict[str, str]], Path]*, *\*\*kwargs: Any*) → Any

    - Run inference on the quantized model for quick accuracy checks post-quantization.

- Parameters

    - - **stage\_output** – Output from the quantization stage.
- **prompt** – Input text to generate from. Can be a string, a Path to a text file,
or a list of chat messages (in which case the last message content is used).
- **\*\*kwargs** – Must include `sequence_length` and `context_length`.
Additional kwargs (e.g. `max_new_tokens`) forwarded to the generator.

- generate\_test\_vectors(*stage\_output: QuantizationOutput*, *config: QuantizationConfig*, *test\_vector\_config: Optional[Union[[TestVectorConfig](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-common.html#qairt.experimental.pipeline.torch.common.configs.TestVectorConfig), Dict[str, Any]]] = None*) → str

    - Record golden test vectors from this stage’s quantized model.

Callable whether the stage just executed or was restored from cache: on a
cache hit `stage_output.quantsim` is `None`, but the restored model
still carries its encodings as registered parameters, so a minimal sim
surface over it is sufficient.

- Parameters

    - - **stage\_output** – Output from the quantization stage.
- **config** – Stage configuration carrying `test_vector_config`.
- **test\_vector\_config** – Overrides the `test_vector_config` on the stage
config when provided. Its `output_dir` field is required and
determines where the vectors land. Accepts a `TestVectorConfig`
instance or a plain dict that will be coerced to one.

- Returns

    - The directory the vectors were written to.

- Raises

    - - **ValueError** – If no `test_vector_config` is available, or the model or
    generator needed to record vectors is missing.
- **RuntimeError** – If no vectors were written (empty dataloader, or every
    sample too short for the requested `slice_num`).

- *classmethod* get\_stage\_dependency() → [StageDependencies](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageDependencies)

    - Declare dependencies for this stage.
Override in subclasses; default is no dependencies.

- *classmethod* load\_from\_cache(*cache\_path: Path*, *\*\*kwargs: Any*) → QuantizationOutput

    - Restore a previously exported quantization output from `cache_path`.

## QuantizationOptStage

Quantization Optimization Stage

Applies a pre-quantization optimization recipe to a loaded model to improve
quantization quality.  The technique is selected by `recipe_name` and
dispatched through the same `RecipeRegistry` used by `QuantizationStage`.

- Supported recipes (registered in `defaults.py`):
    - - seq\_mse\_opt   : SeqMSE weight optimization
- spinquant     : SpinQuant Hadamard rotation
- ada\_scale\_hf  : AdaScale HF-adapter optimization
- gptaq\_hf      : GPTAQ HF-adapter weight quantization
- prefix\_quant  : Prefix KV-cache pre-computation

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.quantization\_opt\_stage.QuantizationOptConfig

    - Bases: [`StageConfig`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageConfig)

Configuration for the quantization optimization stage.

- recipe\_name

    - Registry key for the optimization recipe to run in legacy
mode (e.g. `"spinquant"`, `"ada_scale_hf"`).  Mutually exclusive
with `steps` — provide one or the other, not both.

- steps

    - Composed-mode pipeline — an ordered list of `{name, params}`
entries.  Ignored in legacy mode.

- context\_length

    - Max context length forwarded to the recipe.
Falls back to `generator_config`, then to `DEFAULT_CONTEXT_LENGTH`.

- sequence\_length

    - Sequence length forwarded to the recipe.
Falls back to `generator_config`, then to `DEFAULT_SEQUENCE_LENGTH`.

- dataset

    - Calibration data source.  Accepts either a pre-built
`torch.utils.data.Dataset` or a declarative `dict` spec —
in either case the stage wraps it into a
`torch.utils.data.DataLoader` (`batch_size=1` /
`shuffle=True` / `num_workers=0` / `default_data_collator`)
at `_execute` time.  The DataLoader is a fixed implementation
detail owned by the stage; users provide *data*, the stage owns
iteration.

The dict spec **only reserves one key**:

- `name` (required): dataset registry key
(`"wikitext"` / `"c4"` / `"sharegpt"` / `"json"` / …).

**Every other key is forwarded verbatim to the dataset’s``.load(\*\*kwargs)`` call**, so common inputs like `split`,
`num_samples`, `path`, `apply_chat_template`, etc. just live
at the top level.  Example:

dataset:
      name: sharegpt
      split: train
      num_samples: 1024   # forwarded to ShareGPT.load(...)
    Copy to clipboard

- optimizer\_kwargs

    - Kwargs forwarded verbatim to the recipe’s `apply()`
in legacy mode.

- dsp\_arch

    - DSP architecture selecting the HTP quantsim config (e.g. `"v81"`).
Filled from `soc_details` when unset so it participates in the cache key.

- *field* context\_length*: Optional[int]*  *= None*

    - 

- *field* dataset*: Optional[Any]*  *= None*

    - 

- *field* dsp\_arch*: Optional[str]*  *= None*

    - 

- *field* optimizer\_kwargs*: Dict[str, Any]*  *[Optional]*

    - 

- *field* recipe\_name*: Optional[str]*  *= None*

    - 

- *field* sequence\_length*: Optional[int]*  *= None*

    - 

- *field* steps*: Optional[List[Dict[str, Any]]]*  *= None*

    - 

- model\_post\_init(*context: Any*, */*) → None

    - We need to both initialize private attributes and call the user-defined model\_post\_init
method.

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.quantization\_opt\_stage.QuantizationOptInput

    - Bases: [`StageInput`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageInput)

Upstream model provided by the model loading stage.

- *field* config*: Optional[Any]*  *= None*

    - 

- *field* model*: torch.nn.Module*  *[Required]*

    - 

- *field* tokenizer*: Any | None*  *= None*

    -

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.quantization\_opt\_stage.QuantizationOptOutput

    - Bases: [`StageOutput`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageOutput)

Output from the quantization optimization stage.

- *field* config*: Optional[Any]*  *= None*

    - 

- *field* model*: torch.nn.Module*  *[Required]*

    - 

- *field* prepared\_model*: Optional[torch.nn.Module]*  *= None*

    - 

- *field* quantsim*: Optional[Any]*  *= None*

    - 

- *field* tokenizer*: Any | None*  *= None*

    -

- *class* qairt.experimental.pipeline.torch.llm.stages.quantization\_opt\_stage.QuantizationOptStage

    - Bases: [`Stage`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.Stage)[`QuantizationOptInput`, `QuantizationOptConfig`, `QuantizationOptOutput`]

Applies a pre-quantization optimization recipe to a loaded model..

- Config

    - alias of `QuantizationOptConfig`

- Input

    - alias of `QuantizationOptInput`

- Output

    - alias of `QuantizationOptOutput`

- evaluate(*stage\_output: QuantizationOptOutput*, *config: QuantizationOptConfig*, *\*\*kwargs: Any*) → Dict[str, float]

    - Run the configured evaluation metrics on the quantization optimized model.

- Parameters

    - - **stage\_output** – Output from the quantization stage (model, tokenizer, config,
and optionally a pre-wrapped generator).
- **config** – Stage configuration carrying `evaluator_config` and, via the
pipeline context, the `cache_dir` default output location.
- **\*\*kwargs** – Only `evaluator_config` is accepted, to override the one on the
stage/pipeline config; it takes precedence when provided. Any other
keyword argument raises `TypeError` (guards against silent typos).

- Returns

    - The `{display_name: score}` mapping from `run_evaluation()`.

- Raises

    - - **ValueError** – If no `evaluator_config` with metrics is configured, or the
    model/tokenizer is missing.
- **TypeError** – If an unexpected keyword argument is passed.

- *classmethod* export(*stage\_output: QuantizationOptOutput*, *output\_path: Path*, *\*\*kwargs: Any*) → None

    - Export the optimized model.

HuggingFace models (e.g. from ada\_scale\_hf, gptaq\_hf, prefix\_quant) are saved
in HF format via `save_pretrained`.  Plain torch modules (e.g. SpinQuant
`prepared_model`, SeqMSE `quantsim.model`) are saved as a state-dict via
`torch.save` because they do not carry HF config metadata.

- *classmethod* get\_stage\_dependency() → [StageDependencies](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageDependencies)

    - Declare dependencies for this stage.
Override in subclasses; default is no dependencies.

- *classmethod* load\_from\_cache(*cache\_path: Path*, *\*\*kwargs: Any*) → QuantizationOptOutput

    - Restore a previously exported optimized model from `cache_path`.

## GenAIBuilderStage

GenAI Builder Stage

Builds an `LLMContainer` from ONNX/model artifacts using `GenAIBuilderFactory`.
The stage can also run generation by creating a `T2TExecutor` from the container.

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.gen\_ai\_builder.builder\_stage.GenAIBuilderConfig

    - Bases: [`StageConfig`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageConfig)

Stage config for container build: backend/targets and advanced builder options.

- Validators

    - - `_coerce_compile_config` » `compile_config`
- `_coerce_transform_config` » `transform_config`
- `_validate_compile_args` » `all fields`
- `_validate_max_workers` » `max_workers`
- `_validate_transform_args` » `all fields`

- *field* attach\_model\_for\_arn*: Optional[List[Tuple[int, str, Optional[str]]]]*  *= None*

    - Attach secondary AR models: list of (arn, model\_path, encodings\_path) tuples.

- Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* calibration\_options*: Optional[dict[str, Any]]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* chat\_template*: Optional[str]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* compile\_backend\_extensions\_path*: Optional[FilePath]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* compile\_config*: Optional[[CompileConfig](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-api-compiler.html#qairt.api.compiler.config.CompileConfig)]*  *= None*

    - - Validated by

    - - `_coerce_compile_config`
- `_validate_compile_args`
- `_validate_transform_args`

- *field* compile\_options*: Optional[dict[str, Any]]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* conversion\_options*: Optional[dict[str, Any]]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* max\_workers*: Optional[int]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_max_workers`
- `_validate_transform_args`

- *field* multi\_graph*: Optional[bool]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* native\_kv*: Optional[bool]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* parallel\_build*: Optional[bool]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* prepare\_embedding\_lut*: Optional[bool]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* speculative\_config*: Optional[LadeBuilderConfig | EagletBuilderConfig | SsdBuilderConfig]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* transform\_config*: Optional[ModelTransformerConfig]*  *= None*

    - - Validated by

    - - `_coerce_transform_config`
- `_validate_compile_args`
- `_validate_transform_args`

- *field* transform\_options*: Optional[dict[str, Any]]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- *field* weight\_sharing*: Optional[bool]*  *= None*

    - - Validated by

    - - `_validate_compile_args`
- `_validate_transform_args`

- model\_post\_init(*context: Any*, */*) → None

    - We need to both initialize private attributes and call the user-defined model\_post\_init
method.

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.gen\_ai\_builder.builder\_stage.GenAIBuilderInput

    - Bases: [`StageInput`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageInput)

Upstream model artifact paths provided by the immediate previous stage.

- *field* config\_path*: Optional[str]*  *= None*

    - 

- *field* dm\_encodings\_path*: Optional[str]*  *= None*

    - Draft model encodings path for speculative decoding (wired from DM quantization output).

- *field* dm\_model\_path*: Optional[str]*  *= None*

    - Draft model ONNX path for speculative decoding (wired from DM quantization output).

- *field* draft\_token\_map\_path*: Optional[str]*  *= None*

    - Draft token map JSON path for speculative decoding (wired from DM quantization output).

- *field* lora\_builder\_config*: Optional[[LoraBuilderInputConfig](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-gen-ai-modules-lora.html#qairt.modules.lora.lora_config.LoraBuilderInputConfig)]*  *= None*

    - LoRA builder configuration wired from QuantizationOutput; None when LoRA is disabled.

- *field* model\_path*: str*  *[Required]*

    - 

- *field* preconfigured\_builder*: Optional[[GenAIBuilderHTP](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-gen-ai-api-builders-htp.html#qairt.gen_ai_api.builders.gen_ai_builder_htp.GenAIBuilderHTP)]*  *= None*

    - 

- *field* tokenizer\_path*: Optional[str]*  *= None*

    -

- *pydantic model* qairt.experimental.pipeline.torch.llm.stages.gen\_ai\_builder.builder\_stage.GenAIBuilderOutput

    - Bases: [`StageOutput`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageOutput)

Build output containing the LLM container.

- *field* container*: [LLMContainer](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-gen-ai-api-containers.html#qairt.gen_ai_api.containers.llm_container.LLMContainer)*  *[Required]*

    -

- *class* qairt.experimental.pipeline.torch.llm.stages.gen\_ai\_builder.builder\_stage.GenAIBuilderStage

    - Bases: [`Stage`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.Stage)[`GenAIBuilderInput`, `GenAIBuilderConfig`, `GenAIBuilderOutput`]

Builds an LLMContainer via GenAIBuilderFactory and can run generation through its executor.

- Config

    - alias of `GenAIBuilderConfig`

- Input

    - alias of `GenAIBuilderInput`

- Output

    - alias of `GenAIBuilderOutput`

- artifact\_dirname(*config: GenAIBuilderConfig*, *key: str*) → str

    - Return a human-readable directory name: `<model_short>_container_<hash8>`.

- evaluate(*stage\_output: GenAIBuilderOutput*, *config: GenAIBuilderConfig*, *\*\*kwargs: Any*) → Dict[str, float]

    - Run on-device perplexity evaluation via the Genie Accuracy API.

Requires a device that routes to non-native execution (the Genie Accuracy
API is only reachable through the on-device `GenieAppRunner`). Supported
`evaluator_config.metrics` entries: `{"name": "perplexity",
"dataset_name": str, "dataset_path": str, "engine_convert_to_basic": bool,
"engine_role": str}` — all fields besides `name` are optional.

Note

`context_length` is a no-op here — on-device context length is fixed
at export time. `output_dir` only controls where the intermediate
dataset file is written, not a scores report.

- Parameters

    - - **stage\_output** – Output from this stage, providing the built container.
- **config** – Stage configuration carrying `evaluator_config`.
- **\*\*kwargs** – `evaluator_config` (optional override), `device`
(required), `qairt_sdk_root` (optional), `clean_up` (optional,
default `True`), `prepare_environment` (optional, default
`True`; set `False` to reuse an environment already prepared by a
prior `generate()`/`evaluate()` on the same device session).

- Returns

    - A `{metric_key: score}` mapping where `metric_key` is
`f"{metric_name}_{dataset_name}"`. Metrics that would collide on the
same key are disambiguated with a `_2`, `_3`, … suffix in
configuration order so no score is dropped.

- Raises

    - - **ValueError** – If no evaluator\_config/metrics, no device, an unsupported
    metric name, or an empty `dataset_path` is configured.
- **TypeError** – If unexpected kwargs are passed.
- **NotImplementedError** – If the selected runner does not support
    on-device perplexity evaluation (e.g. native execution).

- *classmethod* export(*stage\_output: GenAIBuilderOutput*, *output\_path: Path*, *\*\*kwargs: Any*) → None

    - Export the built container artifacts to `output_path`.

Uses `container.save()` so the layout is compatible with
`LLMContainer.load()` in `load_from_cache()`.  Device deployment
(`container.export()` with a `target_device`) is a separate step
outside the pipeline export path.

- generate(*stage\_output: GenAIBuilderOutput*, *config: GenAIBuilderConfig*, *prompt: Union[str, List[Dict[str, str]], Path]*, *\**, *lora\_config: Optional[[UseCaseRunConfig](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-gen-ai-modules-lora.html#qairt.modules.lora.lora_config.UseCaseRunConfig)] = None*, *generation\_config: Optional[GenerationConfig] = None*, *\*\*kwargs: Any*) → TextGenerationResult

    - Create a GenAI executor from the container and run text generation.

- Supported `kwargs`:
    - device (Device | None): Target device for execution.
clean\_up (bool): Clean temporary runtime artifacts after execution. Default `True`.
prepare\_environment (bool): Prepare runtime environment before execution. Default `True`.
qairt\_sdk\_root (str | None): Optional QAIRT SDK root override.

- *classmethod* get\_stage\_dependency() → [StageDependencies](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-base.html#qairt.experimental.pipeline.torch.common.bases.stage.StageDependencies)

    - Declare dependencies for this stage.
Override in subclasses; default is no dependencies.

- *classmethod* load\_from\_cache(*cache\_path: Path*, *\*\*kwargs: Any*) → GenAIBuilderOutput

    - Restore a previously exported container from `cache_path`.

## Model Loading Utilities

- *class* qairt.experimental.pipeline.torch.llm.loader.auto\_classes.QcAutoModelForCausalLM

    - Bases: `_QcAutoModelBase`

Qc-enhanced `AutoModelForCausalLM` with internal re-authoring.

Loads text-only (causal LM) checkpoints. For multimodal (image-text-to-text)
checkpoints such as Qwen3.5-VL, use [`QcAutoModelForImageTextToText`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-stages.html#qairt.experimental.pipeline.torch.llm.loader.auto_classes.QcAutoModelForImageTextToText).

The class automatically applies:
1. Module replacement (HF modules → Qc-optimized modules)
2. DynamicLayer patching for KV cache management
3. Config flag setting (use\_cache, padding\_side, etc.)

Example:

# Load with automatic re-authoring (default)
    model = QcAutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B")
    
    # Usage with Qc config
    qc_config = QcAutoConfig.from_model_type(
        "llama",
        base_config,  # obtained separately
        transposed_key_cache=True,
        perform_scatter_kv_cache_update=True,
    )
    model = QcAutoModelForCausalLM.from_pretrained(
        "meta-llama/Llama-3.2-1B-Instruct",
        qc_config=qc_config,
    )
    
    # Load without re-authoring (for debugging)
    model = QcAutoModelForCausalLM.from_pretrained(
        "meta-llama/Llama-3.2-1B",
        model_reauthoring=False,
    )
    Copy to clipboard

- *class* qairt.experimental.pipeline.torch.llm.loader.auto\_classes.QcAutoModelForImageTextToText

    - Bases: `_QcAutoModelBase`

Qc-enhanced `AutoModelForImageTextToText` with internal re-authoring.
Parallel to [`QcAutoModelForCausalLM`](https://docs.qualcomm.com/doc/80-87189-2/topic/qairt-pipeline-stages.html#qairt.experimental.pipeline.torch.llm.loader.auto_classes.QcAutoModelForCausalLM), but for multimodal (image-text-to-text)
checkpoints such as Qwen3.5-VL.

Example:

model = QcAutoModelForImageTextToText.from_pretrained(
        "Qwen/Qwen3.5-VL-...",
        qc_config=qc_config,
    )
    Copy to clipboard

Last Published: Sep 17, 2026

[Previous Topic
EagletV2Qwen35Config](https://docs.qualcomm.com/bundle/publicresource/80-87189-2/topics/qairt-pipeline-llm.md) [Next Topic
Quantization Recipes](https://docs.qualcomm.com/bundle/publicresource/80-87189-2/topics/qairt-pipeline-quantization.md)