# QAIRT Quantization Specification
| **Document Version** | **QAIRT SDK Release** | **AIMET SDK Release** |
| --- | --- | --- |
| v0.1 (Alpha) | v2.36 | v2.3 |
- [Introduction](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#introduction)
- [Converter quantization workflow](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#converter-quantization-workflow)
- [Quantization inputs](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#quantization-inputs)
- [Quantization processing](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#quantization-processing)
- [Quantization outputs](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#quantization-outputs)
- [Target specific processing](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#target-specific-processing)
- [Handle data movement ops](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#handle-data-movement-ops)
- [Tag super groups](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#tag-super-groups)
- [Select op precision](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#select-op-precision)
- [Insert converts](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#insert-converts)
- [Quantize static tensors](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#quantize-static-tensors)
- [Appendix A: Classification of quantization techniques](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-a-classification-of-quantization-techniques)
- [Appendix B: Quantized model formats](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-b-quantized-model-formats)
- [ONNX](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#onnx)
- [TFLite](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#tflite)
- [TensorFlow](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#tensorflow)
- [Appendix C: JSON schema version 2.0.0](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-c-json-schema-version-2-0-0)
- [Per tensor quantization encoding example](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#per-tensor-quantization-encoding-example)
- [Per channel quantization encoding example](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#per-channel-quantization-encoding-example)
- [Blockwise quantization encoding (BQ) example](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#blockwise-quantization-encoding-bq-example)
- [Low power blockwise quantization (LPBQ) example](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#low-power-blockwise-quantization-lpbq-example)
- [Appendix D: Super groups](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-d-super-groups)
- [HTP super groups](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#htp-super-groups)
- [Appendix E: Data movement ops](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-e-data-movement-ops)
- [HTP data movement ops](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#htp-data-movement-ops)
## [Introduction](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id5)
Most neural network models use 32-bit floating point representation for
weights and activations. However, these high-precision representations
are often optimized for inference, especially when deploying models on
edge devices where available memory and compute resources are more
limited. Quantization is a model compression technique used to lower
memory footprint, power consumption, data transfers and improve
inference speed by reducing the precision of the model’s weights and
activations with minimal accuracy loss. For more information on
quantization techniques refer to the Appendix of this document.
The goals of this document are:
- To describe the QAIRT workflow for quantized networks and explain how
quantization data is processed by the QAIRT stack.
- To document the quantization encoding formats supported by the QAIRT
stack for benefit of customers using third-party quantization
tools. Note that Qualcomm’s AI Model Efficiency Toolkit (AIMET)
supports these output formats for seamless operation with QAIRT.
## [Converter quantization workflow](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id6)
Quantization workflow
The above figure shows the high-level workflow for converting from framework
model format to a Deep Learning Contained (DLC). The *qairt-converter*
converts a model for source framework format into a DLC which can be used for inference.
### [Quantization inputs](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id7)
The *qairt-converter* accepts the following inputs:
- **Model input (mandatory)**
- **Float model:** A 32-bit floating point model (no quantization
information)
- **Quantized model:** A model containing quantized nodes. Ex:
TFLite models.
- **Q-DQ model:** A model where quantization information is
signaled using Q-DQ nodes. E.g. an ONNX model with QuantizeLinear,
DequantizeLinear
Note
For more details see [Appendix B: Quantized Model
Formats](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-b-quantized-model-formats)
- **Quantization Encoding Overrides (optional)**
- **Encoding Overrides:** a JSON file with quantization information
for each tensor may be passed using the *–quantization\_overrides*
option. Quantization information in the encodings override file
will take precedence over information in source model file.
See [Quantization
Overrides](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/quantization.html#quantization-overrides)
in the QAIRT SDK documentation for detailed information.
- Note that AIMET can be
used to perform quantization and to produce a quantization
overrides JSON file. Quantization information from third-party
tools can be specified in the same format.
- See [JSON Schema Version 2.0.0](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-c-json-schema-version-2-0-0) for a
detailed description of the encoding file format.
- **Target (optional)**
- The target backend (*HTP, LPAI*, etc) can be specified using
*–target\_backend* option. Run *qairt-converter–help* for additional details on it. When the target device is
provided, additional target-specific processing during the
conversion processing is performed. Without target device
information, only target-agnostic quantization will be performed.
Examples commands:
- Converting a 32-bit floating-point model with quantization
overrides file
`qairt-converter -i model.onnx --quantization_overrides
model.encodings –-target_backend=HTP`
- Converting an ONNX Q-DQ model
`qairt-converter -i model_qdq.onnx –-target_backend=HTP`
### [Quantization processing](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id8)
Quantization processing by the *qairt-converter* consists of two steps:
quantization extraction and quantization adaptation.
The *qairt-converter* first extracts quantization encoding
information from the source model and encoding overrides file (if
present). Note that quantization information in the source model can
exist at different granularities such as per-tensor, per-channel,
per-block, etc… See [Classification of Quantization
Techniques](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-a-classification-of-quantization-techniques) for more information.
If a tensor has encoding information in both the model and
overrides file, the overrides file will take precedence. See [JSON
Schema](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-c-json-schema-version-2-0-0) and [Quantized Model
Formats](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-b-quantized-model-formats) for more details.
Next, target-specific processing is performed, and the resulting
quantization information is applied to tensors. When encodings
associated with an operation cannot be mapped directly to the target
device the encodings are adapted to the specified target. The
adaption process has a defined set of rules that are documented
in [Target Specific Processing](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#target-specific-processing).
### [Quantization outputs](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id9)
**DLC** - The output of the *qairt-converter* is a DLC ready for
inference or compilation with the QAIRT stack.
## [Target specific processing](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id10)
During target-specific processing, the quantization information
extracted from processing the source model and overrides files is
compared with the supported datatypes and operations on the specified
target. If the extracted quantization information cannot be directly
mapped, quantization information is adapted.
The following guidelines are used when adapting an encoding to a target.
- 32-bit floating point tensors are assigned float16 precision (default
precision for the targets supported by QAIRT).
- Tensors with no encoding information are treated as 16-bit floating
point.
- If the extracted quantized encoding if not supported by the target,
the qairt-converter will fall back to a higher precision (that is
supported on the target) in order to maintain accuracy. Precision
precedence order is: float32, float16, int32, int16, int8, int4, and
int2 (from highest to lowest).
The figure below shows the processing steps involved in adapting
quantization encodings for a target. Steps include handling for data
movement ops, supergroups, promotion and assignment of encodings,
inserting appropriate conversion ops, quantizing static tensors, etc…
and are detailed below.
Adapt Encodings – Algorithm
### [Handle data movement ops](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id11)
Assign quantization encoding to the tensors associated with data
movement ops and ensure matching quantization encodings are assigned
along the chain of data movement operations. Operations that do not fall
in the category of data movement ops are unaffected. List of [Data
Movement Ops](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-e-data-movement-ops).
*Rules:*
- Quantization encoding of the input and output tensor of data movement
operations must be equivalent.
- A continuous sequence, or chain, of data movement ops is uniformly
assigned the same encoding value as the range is not expected to
change within the chain.
- When different bit widths are detected at input and output of the op
or chain of ops, the module assigns the minimum bit width for that op
or group. This may result in a ConvertOp or CastOp at the start or
end of the chain.
### [Tag super groups](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id12)
A super group is a subgraph which executes in the same precision from start
to end. Super groups are identified and tagged at this stage. Super
groups are used to:
- Avoid quantization noise by holding intermediate tensor data in
higher precision to improve accuracy.
- Enable targets to fuse the subgraph, when possible, to improve
performance.
- For example, the *Conv->ReLU* pattern will be detected and marked in the DLC so
that fusion can be performed during target compilation.
Super groups are target specific. A complete list of the super groups
can be found in [Super groups](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-d-super-groups). Tools like AIMET
simulate super group patterns during quantization and calibration for
improved accuracy and performance.
*Rules:*
- The super group pattern must be supported by the target ([Super
groups](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#appendix-d-super-groups)).
- Quantization encoding must be only specified for tensors at the start
and end of the super groups, and no encoding must be specified for
intermediate tensors.
- For a sequence of ops to qualify as a super group the intermediate
tensors must not be consumed outside of the super group ops. (For example,
in the *Add->Relu* sequence the tensor between *Add* and *Relu*)
- The super group operator dimensions and other attributes must be
supported by the specified target.
### [Select op precision](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id13)
- Each target supports operations in different precisions (datatype,
bit width, symmetry, etc) as tabulated in the [Backend Supplemental
Op Definition for each
target](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/operations.html#backend-supplements).
Tensors with missing quantization encoding information are
assigned the default, float 16 datatype. For each op, the selection
process determines if the encoding is supported by the target. If the
encoding is supported, then it is used as-is. If the encoded
precision is not supported, the precision will fall back to the
nearest supported higher precision. Precision precedence order is:
float32, float16, int32, int16, int8, int4, and int2 (from highest to
lowest). If a higher precision is not available, it is reported as
an error.
*Rules:*
- Tensors with no encoding are first assigned float16 encoding
(default). At this point all tensors must have an assigned encoding.
- Quantization encoding is assigned to tensors and then ops connected
to that tensor are evaluated to ensure a matching kernel is available
on a backend for execution.
- When a matching kernel is not available, iteratively, look for
nearest higher precision kernel.
- Checks performed when finding a matching a kernel.
1. When the user provided Symmetry is not supported it is an error.
2. When user provided encodings do not meet the granularity
(per-tensor, per-channel or per-block) requirements then it is an
error.
3. When the user provided *y\_scale* does not meet constraint
specified by the target then, it is an error unless it can be
rescaled to match the desired range for the selected bit width.
4. When the user provided sign of the *output\_dtype* does not match
then, a kernel with opposite sign is looked for by changing the
offset value.
5. When the user provided bit width is not supported by the kernel
even after changing the sign then, higher bit width is looked for
by repeating the steps from 1 to 5.
### [Insert converts](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id14)
For every tensor where the tensor’s producer and consumer operations
have different datatype requirements an appropriate Convert (rescale,
quantize, or de-quantize) or Cast Op is inserted between ops.
*Rules:*
- Insert a convert or cast op from tensor datatype present in the
model to a user specified datatype
- If the selected datatype is different from user specified datatype,
then another conversion is done from user specified to selected
datatype.
### [Quantize static tensors](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id15)
Quantize static tensor values and store to avoid inserting
a conversion op. This step helps reduce runtime cycles.
*Rules:*
- Static tensors are quantized using the precision selected in Select
Op Precision. When quantization bit width is less than the 8-bit
multiple elements may be packed in packed into 8-bit. For example:
two 4-bit elements can be packed in one 8-bit datatype.
## [Appendix A: Classification of quantization techniques](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id16)
Quantization can be categorized based on the aspects of the model it
affects and the methods it uses:
- **Based on Scope**
- *Weight Quantization:* Reduces the precision of weights, which
represent learned parameters. Weight quantization is particularly
effective because weights are fixed after training.
- *Activation Quantization:* Reduces the precision of activation,
which represents intermediate outputs of the model during
inference. Activation quantization is more challenging due to
dynamic data ranges.
- *Bias Quantization:* Reduces precision of bias. Quantization of
bias depends on weight and activation quantization.
- **Based on Granularity**
- *Per Tensor Quantization:* The entire tensor (weights or
activations) shares a single scaling factor. [Example
JSON](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#per-tensor-quantization-encoding-example)
- *Per Channel Quantization (PCQ):* Each output channel of a layer
has its own scaling factor. [Example JSON](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#per-channel-quantization-encoding-example)
- *Block Quantization (BQ):* Quantizes weights or activations in
blocks rather than individually or per entire tensor/channel. [Example JSON](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#blockwise-quantization-encoding-bq-example)
- *Low Power Block Quantization (LPBQ):* Encodings at a lower bit
width are determined and then adjusted such that they lie on a
common higher bit width per channel grid.
[Example JSON](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#low-power-blockwise-quantization-lpbq-example)
## [Appendix B: Quantized model formats](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id17)
Quantization information can be provided in the model source graphs
using Q-DQ nodes or using quantized operations depending on the source
framework being used.
The following table lists the combinations of inputs supported by the
*qairt-converter* for each framework.
Input model formats, framework and quantization granularity support matrix
| Framework | Formats | Quantization Granularity |
| --- | --- | --- |
| ONNX | Float model, Q-DQ model | Per-tensor, Per-channel,
Per-block |
| TensorFlow | Float model, FakeQuant model | Per-tensor, Per-channel |
| TFL | Float model, Quantized model | Per-tensor, Per-channel |
| PyTorch | Float model | Per-tensor, Per-channel |
### [ONNX](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id18)
A sequence of *QuantizeLinear*
and *DequantizeLinear* (Q-DQ) nodes are used in the ONNX graph
to specify quantization information. During model conversion
in *qairt-converter*, quantization information from Q-DQ sequence is
extracted as metadata and the
corresponding *QuantizeLinear* and *DequantizeLinear* nodes are folded
to produce a quantized graph.
For example, a ReLU node between a dequantize and quantize node, *int8 →Dequantize → fp32 → ReLU → fp32 → Quantize → int8* is treated equivalent
to a quantized ReLU operator *int8-> ReLU → int8*. First, quantization
encoding from Q-DQ nodes is extracted and stored as metadata (also
referred to as Encoding) attached to the corresponding tensor. Next, the
encoding is applied to the tensor resulting in a quantized graph.
The following figure shows how the Q-DQ nodes are
folded to extract quantization encoding and then produce a quantized
graph. Encoding collected from the source model may be overridden
using the *–quantization-overrides* option.
Extract encodings from Q-DQ model and quantized tensors to build a quantized graph
For more information about ONNX Q-DQ, see [QuantizeLinear - ONNX
1.18.0
documentation](https://onnx.ai/onnx/operators/onnx__QuantizeLinear.html).
### [TFLite](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id19)
TFLite uses quantized nodes to convert floating-point models to lower
precision. The TFLite model, including its quantized nodes, is converted
into a DLC. This process involves deriving quantization encodings using
TFLite quantization parameters (scale, zero-point) before associating it
with the tensor in the DLC. Encodings collected from the source model
may be overridden using the *–quantization-overrides* option.
### [TensorFlow](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id20)
The *FakeQuant* node in TensorFlow is used for quantization-aware
training. It simulates the effects of quantization during training,
allowing the model to learn the quantization effects and potentially
achieve higher accuracy when quantized. Quantization information present
in the *FakeQuant* node stored as quantization encoding metadata (preceding figure
shows int8 encodings stored as metadata associated with float-16 tensors).
Encoding collected from the source model may be overridden
using the *–quantization-overrides* option. For more information on
FakeQuant see
[tf.quantization.fake_quant_with_min_max_args](https://www.tensorflow.org/api_docs/python/tf/quantization/fake_quant_with_min_max_args).
## [Appendix C: JSON schema version 2.0.0](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id21)
JSON schema uses keys which are closely aligned with attributes of the
ONNX *QuantizeLinear* operators detailed below.
- **name**: name of the tensor
- **output\_dtype:** tensor data type with bit-width appended. Valid
values for this field are listed. “int4”, “uint4”, “int8”, “uint8”,
“int16”, “uint16”, “int32”, “uint32”, “float32”, “float16”.
- **y\_scale**: tensor data is quantized with this scale value.
- **y\_zero\_point (optional, default = 0)**: tensor data is quantized
with the provided zero point. Internally this is stored as *offset =- y\_zero\_point*. Its value may be used to interpret symmetry of the
quantized data.
- **axis (optional)**: Set when per-channel quantization or block
quantization is used. For per-channel quantization, it indicates the
channel axis along which the scale values are specified. For block
quantization, (when *block\_size* is specified) it indicates the
tensor axis along which blocking is done.
- **block\_size (optional)**: Set only when block quantization is used.
Size of the block along the specified *axis*.
- **per\_channel\_float\_scale (optional):** TBD
- **per\_block\_int\_scale (optional)**: Applicable when Low Power Block Quantization is used,
for which block\_size must be provided. An integer scale is assigned per block.
### [Per tensor quantization encoding example](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id22)
{
“name”: “activation”,
“output_dtype”: “uint8”,
“y_scale”: 1.0,
“y_zero_point”: 128
}
Copy to clipboard
### [Per channel quantization encoding example](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id23)
Example encodings for channel *axis = 0* and *length = 3.*
{
“name”: “conv.weight”,
“output_dtype”: “int8”,
“y_scale”: [
0.0050100767984986305,
0.0017133733490481973,
0.0017133733490481973,
]
“axis”: 0
}
Copy to clipboard
### [Blockwise quantization encoding (BQ) example](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id24)
shape: [3, 64]
channel\_axis: 0
block\_axis: 1
block\_size: 32
{
“name”: “conv22.weight”,
“output_dtype”: “int4”
“y_scale”: [
[0.01, 0.02],
[0.03, 0.04],
[0.05, 0.06]
],
“y_zero_point”: [
[0, 0],
[0, 0],
[0, 0]
],
“axis”: 1,
“block_size”: 32,
}
Copy to clipboard
### [Low power blockwise quantization (LPBQ) example](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id25)
{
“name”: “conv2.weight”,
“output_dtype”: “int4”
“per_channel_float_scale”: [
0.01, 0.02, 0.03
],
“per_block_int_scale”: [
[1, 2, 16],
[2, 16, 7],
[16, 3, 1]
],
“offset”: 0,
“block_size”: [1, 3],
}
Copy to clipboard
## [Appendix D: Super groups](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id26)
Super groups are listed per target in this section.
### [HTP super groups](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id27)
The following table lists the super groups for HTP backend. These super
groups are supported by Hexagon architecture version greater than v73.
| **Super group patterns** |
| --- |
| QNN\_OP\_ELEMENT\_WISE\_ADD + QNN\_OP\_RELU |
| QNN\_OP\_CONV\_2D + QNN\_OP\_HARD\_SWISH |
| QNN\_OP\_CONV\_2D + QNN\_OP\_PRELU |
| QNN\_OP\_CONV\_2D + QNN\_OP\_RELU\_MIN\_MAX |
| QNN\_OP\_CONV\_2D + QNN\_OP\_RELU |
| QNN\_OP\_TRANSPOSE\_CONV\_2D + QNN\_OP\_RELU |
| QNN\_OP\_FULLY\_CONNECTED + QNN\_OP\_RELU |
## [Appendix E: Data movement ops](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id28)
Data movement ops must have the same input and output quantization
encodings.
### [HTP data movement ops](https://docs.qualcomm.com/doc/80-63442-50/topic/applyencodings.html#id29)
The following is the complete list of data movement ops supported by HTP.
| **Data Movement Ops** |
| --- |
| QNN\_OP\_GATHER |
| QNN\_OP\_GATHER\_ELEMENTS |
| QNN\_OP\_GATHER\_ND |
| QNN\_OP\_TOP\_K |
| QNN\_OP\_BATCH\_TO\_SPACE |
| QNN\_OP\_CHANNEL\_SHUFFLE |
| QNN\_OP\_SPACE\_TO\_BATCH |
| QNN\_OP\_TILE |
Last Published: Oct 10, 2025
[Previous Topic
Additional details](https://docs.qualcomm.com/bundle/publicresource/80-63442-50/topics/quantization.md) [Next Topic
Tutorials](https://docs.qualcomm.com/bundle/publicresource/80-63442-50/topics/tutorials.md)