# Operator and data type support

## Operator support

This section describes the operator support available for the device across various frameworks.

To determine the list of operators supported by the device for various frameworks, you can run the following command

Example command to generate operators supported for onnx
    
    /opt/qti-aic/exec/qaic-compile -operators-supported=onnx
    Copy to clipboard

This command generators a file `OnnxSupportedOperators.txt` which comprehensive list of ops supported.
It’s important to note that the operator support keeps expanding with the release of new SDK versions.

Note

-operators-supported supports only onnx, tensorflow, pytorch.
    Copy to clipboard

Note

`onnx` is the preferred format to compile the model for the device.

### Handling unsupported operators

Sometimes you might encounter errors related to unsupported operations while compiling the model for the device.

For instance, certain operations like einsum present in the model file might not be directly supported by the device.

In such scenarios, the [Model Preparator](https://docs.qualcomm.com/doc/80-99100-3/topic/index_Prepare-the-model.html#reference-to-model-preparator-tool) tool can be employed to modify the model and substitute these unsupported operations with their corresponding mathematical equivalent subgraphs.

# Data type support

The Cloud AI SDK supports multiple numeric formats,
each offering a different balance between model accuracy, memory
footprint, and inference throughput. The format you choose
directly affects both output quality and hardware performance.

The following sections describe each supported format and its use case.

## FP32 (single precision floating point)

Models can be executed in FP32 for use cases where accuracy is critical
and computational efficiency isn’t a primary concern. It’s essential
to note that FP32 models tend to have larger sizes and will exhibit
lower throughput performance. FP32 execution is supported but not
recommended.

## FP16 (half precision floating point)

FP16 strikes a balance between accuracy and efficiency, making it
suitable for most deep learning workloads.

If a model is originally trained in FP32 format, it can be
down-converted to FP16 during the compilation process using the
`-convert-to-fp16` flag. However, certain scenarios may involve
constants beyond the FP16 range. In such cases, it’s recommended to
clip values to the FP16 range (as demonstrated in the fix\_onnx\_fp16
function in the NLP tutorials in [Cloud-ai-sdk
repo](https://github.com/quic/cloud-ai-sdk)).

## FP8 (8-bit floating point)

Models in FP8 format are supported through [Qualcomm Efficient-Transformers](https://github.com/quic/efficient-transformers).

## Shared micro-exponents (narrow precision format)

See the
[Open Compute Project OCP Microscaling Formats (MX) Specification](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf) for more information about shared micro-exponents. The compiler currently supports MXFP6 with planned support for all of the formats.

Models compiled in MXFP6 format store FP32/FP16 weights using a 6-bit
format. By doing so, it significantly reduces model sizes due to the
compact representation. This format is particularly beneficial for
models that require high data bandwidth, such as large language models
(LLMs).

AI 100 stores matmul weights in MXFP6 format while keeping the rest of
the weights in FP16 format. Computation/activations on the NSP still
occur in FP16.

LLMs experience up to 2x throughput with minimal accuracy loss with
MXFP6 format. Within a constant memory footprint, a larger model can be
supported with MXFP6.

FP32 models can be compiled into MXFP6 format using the compiler flag
`-mxfp6-matmul`. FP16 execution should use both `-convert-to-fp16`
and `-mxfp6-matmul` flags for the `qaic-compile` compiler CLI.

## INT8 (8-bit integer)

AI 100 supports INT8 quantized models, especially relevant
for Natural Language Processing (NLP) and Computer Vision (CV) tasks.

Quantization Methods Supported:

- Quantization Schema for Weights and Activations: Both symmetric and asymmetric.
- Quantization Calibration: Options include KLMinimization, KLMinimizationV2, MSE, SQNR, and
Percentile (with percentile calibration values: 99.9, 99.99, 99.999,
99.9999).

The SDKs provide tools to run a profile-guided quantization (PGQ) sweep, allowing you to identify the
optimal quantization parameters for your specific requirements.

## INT4 (4-bit integer)

AWQ and GPTQ quantized models with 4-bit weights are supported
through [Qualcomm Efficient-Transformers](https://github.com/quic/efficient-transformers)

## BF16 (BFloat16)

If a model is trained in BF16 (bfloat16), ensure that the weights are
scaled down using an appropriate scaling factor to prevent
intermediate activations from overflowing into `fp16`. Qualcomm can
provide a script to identify the scaling factors to scale down the
weights of the models.

Last Published: Aug 25, 2026

[Previous Topic
Introduction to the model preparator tool](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/index_Prepare-the-model.md) [Next Topic
Compile the model](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/index_model-compilation.md)