# 11 End-to-end workflow – QInference Optimizer
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
Deployment of a neural network from development to production requires preparing it for
inference, optimization and fine-tuning. This is a multi-stage and iterative process
involving multiple packages, tools, etc. The flow diagram of training to inference
workflow is explained in [Figure : 1](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html#introduction_qinference_optimizer__fig_idv_vzt_fyb_svenugop_07-25-23-1127-21-602).
Figure : Taking a trained model to production grade compilation
QInference Optimizer is a Python-based tool that serves as a turnkey solution that can
automate this process and provide a final optimized model and parameters to users with
minimal inputs and intervention. Helps users take a model from development to commercial
deployment in few easy steps.
It performs automated evaluation of both accuracy metric and throughput for any given
model and returns the best set of compiler parameters that meet the targets set by the
user.
**Parent Topic:** https://docs.qualcomm.com/doc/80-PT790-993B/topic/dl_inference_tools_part.html
## 11.1 Features
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
1. Provides a fire-and-forget automation to achieve user provided targets
1. Prepares model for inference
2. Performance optimization
3. Accuracy tuning, including quantization and Automatic Mixed Precision
2. Alleviates manual effort and reduces overall tuning time from days to hours
3. Exposes a simple and easy to use API as well as CLI for user to provide their
trained models, datasets and targets.
4. Architecture-specific optimization for Cloud AI 100 backend
1. Selection of quantization / calibration / operator precision setting
2. Shortens accuracy and performance tuning time
5. Accuracy and performance tuning with hardware in loop
6. Scalable design that exploits parallelism if there are multiple AIC cards
available
7. Supports three Execution modes that enables user to evaluate and tune their models
based on their use-case, requirement, targets and time:
1. Default mode – Runs the complete pipeline and tunes for most optimal
accuracy & performance (takes more tuning time)
2. Quick mode – Runs the pipeline with smartly limited configuration parameters
and tunes for reasonable accuracy & performance (shorter tuning time)
3. Expert mode – Runs the pipeline based on user-selected precision, tuning
type and search parameters (tuning time is use case dependent)
8. Upon completion, returns the optimized framework graph, production-grade QPC binary
and a parameter settings file identified from the tuning stages.
**Parent Topic:** [End-to-end workflow – QInference Optimizer](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
## 11.2 Pipeline
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
The QInference Optimizer consists of various functional blocks (see below Figure) which
form a pipeline of the following stages:
1. Initialization
1. Preparation of the ONNX model for optimal AIC inference.
2. Analysis of the model to identify optimization opportunities.
3. Optimize based on model architecture.
2. Execution
1. Accuracy tuning using quantization and Automatic Mixed Precision, if
needed.
2. Performance tuning for throughput or latency.
Figure : Overview and functional blocks of QInference Optimizer
**Parent Topic:** [End-to-end workflow – QInference Optimizer](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
## 11.3 Examples - Jupyter Notebooks
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
Users can refer to the Jupyter notebooks located in the
`/opt/qti-aic/tools/qaic-inference-optimizer/examples` folder to get
started with the QInference Optimizer tool. The following table provides brief
descriptions of each Jupyter notebook.
Table : QInference Optimizer Notebook descriptions
| Notebook name | Description |
| --- | --- |
| QInfOptimizer\_introduction.ipynb | Provides an overview and introduces the environment setup, inputs,
configuration and usage of the QInference optimizer. |
| QInfOptimizer\_resnet50.ipynb | Demonstrates the usage of QInference Optimizer for the resnet50
classification model. Includes steps for fetching datasets, the ONNX
model and running the tool using the different execution modes. |
| QInfOptimizer\_yolov3.ipynb | Demonstrates the usage of QInference Optimizer for tuning the YOLOv3
object detection model. Illustrates the generation of the ONNX model
from open source and evaluation of model using device side NMS
post-processing. |
| QInfOptimizer\_bert\_large.ipynb | Demonstrates the usage of QInference Optimizer for tuning the BERT
Large Natural Language Processing model. Includes steps for fetching
datasets, the ONNX model and running the tool using the default
execution mode. |
**Parent Topic:** [End-to-end workflow – QInference Optimizer](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
## 11.4 Environment Setup
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
This section describes the minimum system requirements and generation of docker image and
container for setting up the execution environment for this tool.
**Parent Topic:** [End-to-end workflow – QInference Optimizer](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
## 11.4.1 Minimum System Requirements
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
- Ubuntu 18.04
- Qualcomm Cloud AI 100 (AIC) device
- Qualcomm AI 100 Software Development Kit >= 1.10
- Platform SDK
- Apps SDK
- QAIC docker
- Sufficient disk space based on model architecture (average 50-60 GB)
- Intel i7 (32 GB RAM)
**Parent Topic:** [Environment Setup](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer-environment-setup.html)
## 11.4.2 Build docker with QInference Optimizer environment
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
The QAIC Apps SDK includes a set of scripts and Dockerfiles for creating a container
environment for the QInference Optimizer tool.
Steps are as follows:
1. Navigate to docker-build directory in the extracted or unzipped Apps
SDK
$ cd /tools/docker-build/Copy to clipboard
2. Generate docker image with QInference Optimizer package along with pytools
package
$ bash build_image.sh \--apps-sdk qaic-apps.zip \--platform-sdk qaic-platform-sdk-x86_64-ubuntu.zip \--os ubuntu18 \--install-qaic-pytools \--install-qinf \--tag qinf_v1.0
Copy to clipboard
3. Create container to start using QInfOptimizer Tookit environment. Recommend
attaching all available AIC devices to the container to aid in faster evaluation of
your
model.
$ docker run -dit --privileged \--name qinf \--device /dev/qaic_aic100_0 \--device /dev/qaic_aic100_1 \ (repeat for multiple devices)qaic-pytools-qinf-ubuntu18-x86_64:qinf_v1.0 bash
Copy to clipboard
4. Attach to created container to get
started.
$ docker attach qinfCopy to clipboard
**Parent Topic:** [Environment Setup](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer-environment-setup.html)
## 11.5 Configuration
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
The parameters required by the tool are configured in a YAML configuration file. The
config file requires mandatory fields like Accuracy target, dataset (Validation &
Calibration), Model path, any input parameters for undefined variables in the model,
mode of tuning and few more. The steps to create the configuration file is detailed
below.
**Parent Topic:** [End-to-end workflow – QInference Optimizer](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
## 11.5.1 Create the configuration YAML file
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
The QInference Optimizer requires a configuration file in the YAML format with the
information about the model, model preparation and accuracy evaluation. This is defined
as 3 sections:
- common\_params
- model\_preparation\_params
- accuracy\_evaluation\_params
**Parent Topic:** [Configuration](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-configuration.html)
## 11.5.1.1 common\_params
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
The common\_params section is necessary to provide information about the model’s input and
output nodes, any ONNX symbols and parameters for compilation and evaluation.
High-level structure of this section:
common_params:
# Mandatory params
inputs_info:
- :
type:
shape: []
outputs_info:
- :
type:
shape: []
# Optional params
batchsize:
compiler_params:
: value
onnx_symbols: # comma delimited string. Syntax: symbol_1=value_1,symbol_2=value_2,..
onnx_symbol_batch:
onnx_custom_op_lib:
Copy to clipboard
Below is the description of the headers for this section:
| Field Name | Description | Type | Default |
| --- | --- | --- | --- |
| `inputs_info` | (Mandatory) Information about the name, type and shape of the model’s
input nodes. | Dict | **-** |
| `outputs_info` | (Mandatory) Information about the name, type and shape of the model’s
output nodes. | Dict | **-** |
| `batchsize` | (Optional) Batchsize to be used for model compilation, performance
and accuracy evaluations. | Integer | 1 |
| `compiler_params` | (Optional) AIC compiler args to be used for model compilation.
Quantization related args are not accepted. Any performance related args
will be used as the max limit for the Performance Evaluation of the
given model. Note that values passed via API takes precedence. | Dictionary | **-** |
| `onnx_symbols` | (Optional) comma-delimited string of onnx symbol name followed by
value to be used. If model has any symbols but values are not provided,
then they would be replaced with value of 1 by default. Syntax:
`symbol_1=value_1,symbol_2=value_2` | String | **-** |
| `onnx_symbol_batch` | (Optional) Name of the onnx symbol used for batch dimension. | String | **-** |
| `onnx_custom_op_lib` | (Optional) Path to ONNX custom op library for usage with
ONNX-runtime. This should be provided if given ONNX model has custom
operators, else evaluation of reference accuracy and Automatic Mixed
Precision would fail.
User can also copy their custom-op library file to
“`/opt/qti-aic/tools/onnxrt-custom-ops”` which
would be automatically loaded during inference. | String | **-** |
**Parent Topic:** [Create the configuration YAML file](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-create-the-configuration-yaml-file.html)
## 11.5.1.2 model\_preparation\_params
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
In this section, based on the model type, user can choose and set params for Object
Detection and Transformer based models.
## Object Detection models
For Object Detection models, user can choose to modify the model and perform NMS post
processing on the device using QAic QDetect layers (refer to [QAic QDetect layers](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic_qdetect_layers_introduction.html)). To enable this, user should set
nms\_post\_processing to device and provide parameters related to NMS. Below table
lists and describes these parameters.
Below is the description of the headers for this section:
| Field Name | Description | Type |
| --- | --- | --- |
| `nms_post_processing` | (Optional)``Field to denote if user wants to modify the
model for 'device' side post-processing for OD models. If it is
set to device, below params must be provided. | String |
| `max_output_size_per_class` | Maximum number of boxes to be selected by non-max suppression per
class. | Integer |
| `max_total_size` | Maximum number of boxes to output from the model. | Integer |
| `iou_threshold` | IoU threshold for NMS-related computation. | Float |
| `score_threshold` | Score threshold for NMS-related computation. Boxes whose score
value is less than this will be discarded. | Float |
| `clip_boxes` | A Boolean flag that indicates whether to clip the resultant boxes
between [0,1] or not. | Boolean |
| `pad_per_class` | A Boolean flag that indicates whether to pad the resultant boxes.
If false, the resultant boxes are padded to max\_total\_size. If true,
the resultant boxes are padded to be of length
max\_output\_size\_per\_class \* num\_classes, unless it exceeds
max\_total\_size, in which case it is clipped to
max\_total\_size. | Boolean |
Alternatively, user can use a partitioned model and perform NMS processing on the
host using QAic Smart NMS application (refer to [QAic Smart NMS](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-introduction.html)). Furthermore, for such cases, user
can choose the qaic\_smart\_nms plugin for processing the SmartNMS outputs (refer to
[Inbuilt postprocessor plug-ins](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Inbuilt-post-processor-plugins.html)) .
## Transformer Based Models
- User has to specify if the model is using "packing strategy" for inputs
through transformer\_packed\_strategy param. Packing strategy for inputs means
that we are "packing" together multiple inputs into a single input.
For example, inputs of sequence length 80,100, and 120 can be "packed"
together to form a single input of sequence length 80+100+120 = 300.
- By default, transformer\_packed\_strategy param is False which means our tool
considers the model as unpacked.
| Field Name | Description | Type |
| --- | --- | --- |
| transformer\_packed\_strategy
` ` | (Optional) Field to denote if the model is packed or unpacked. By
default, the param is set to False. | Boolean |
High-level structure of this section:
model_preparation_params:
transformer_packed_strategy:
nms_post_processing:
nms_params: # Required when nms_post_processing is set to device.
max_output_size_per_class:
max_total_size:
iou_threshold:
score_threshold:
clip_boxes:
pad_per_class:
Copy to clipboard
**Parent Topic:** [Create the configuration YAML file](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-create-the-configuration-yaml-file.html)
## 11.5.1.3 accuracy\_evaluation\_params
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
This is a mandatory header which would be used for processing the dataset, the model’s
outputs and compute the model’s accuracy metric. Refer to [Accuracy tune](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Accuracy-Tune.html) for details on configurating the
dataset, processing and evaluator sections
High-level structure of this section:
accuracy_evaluation_params:
dataset:
transformation:
- plugin: <...>
processing:
preprocessing:
transformations:
- plugin: <...>
- plugin: <...>
postprocessing:
transformations:
- plugin: <...>
- plugin: <...>
evaluator:
metrics:
- plugin: <...>
Copy to clipboard
**Parent Topic:** [Create the configuration YAML file](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-create-the-configuration-yaml-file.html)
## 11.6 Usage
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
QInference Optimizer allows usage via API and CLI.
Note: Configure the below ulimit and
environment variable before running QInference optimizer via API or CLI. This is
according to the steps mentioned in Section [Accuracy evaluator usage](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Accuracy-Evaluator-usage.html).
$ ulimit -n 20480
$ export OMP_NUM_THREADS=1
Copy to clipboard
For API, we have an initialization step followed by an execution step.
**Parent Topic:** [End-to-end workflow – QInference Optimizer](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
## 11.6.1 API
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
For API usage, user should initialization the QInference Optimizer and then use the
execution API to run the pipeline. We have separate APIs for each of the 3 execution
modes which is described in the following sections.
**Parent Topic:** [Usage](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-usage.html)
## 11.6.1.1 Initialization
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
Note: Ensure to import QInfOptimizer
before any other required imports
An example of initializing the QInference optimizer is below:
from qinf.api.qinf_optimizer import QInfOptimizer
from qinf.api import ModelArch
# Initializing the QInference Optimizer
qinf = QInfOptimizer(
config='mv2_config.yaml',
model='mobilenetv2.onnx',
model_arch=ModelArch.MOBILENETV2,
work_dir='qinf_temp',
aic_device_list=[0])
Copy to clipboard
Description of the initialization API arguments are below:
| Field Name | Description | Type | Default |
| --- | --- | --- | --- |
| `config` | Configuration YAML file with common\_params, model\_preparation\_params
and accuracy\_evaluation\_params. | String | **Mandatory** |
| `model` | The path to ONNX model file | String | **Mandatory** |
| `model_arch` | Optional parameter indicating the architecture of the model, e.g.,
bert, yolov5. If not provided, the tool will try to identify and use
accordingly.
**Suggest setting this to ModelArch.NLP for transformer-based models.**
Refer to Appendix for list of supported model-architectures | ModelArch | ModelArch.UNKNOWN |
| `work_dir` | Path to directory for storing intermediate artifacts and
results | String | qinf\_temp |
| `onnx_symbols` | ONNXSymbols dataclass object for indicating the model’s symbols and
the symbol for batch. This information can also be passed via the config
file, as shown in the example above. Note that value passed via API
takes precedence. Example:
ONNXSymbols(onnx_symbols={'sl': 1},
onnx_symbol_batch='batchsize')Copy to clipboard | ONNXSymbols | **-** |
| `onnx_custom_op_lib` | Path to ONNX custom op library for usage with ONNX-runtime. This
should be provided if given ONNX model has custom operators, else
evaluation of reference accuracy and Automatic Mixed Precision would
fail.
User can also copy their custom-op library file to
“`/opt/qti-aic/tools/onnxrt-custom-ops”`” which
would be automatically loaded during inference. | String | **-** |
| `compiler_params` | AIC compiler args to be used for model compilation. Quantization
related args are not accepted. Any performance related args will be
used as the max limit for the Performance Evaluation of the given
model.
Example:
{'aic-num-cores': 1, 'aic-preproc': True, 'multicast-weights': True}Copy to clipboard | Dictionary | **-** |
| `aic_device_list` | List of AIC device IDs to use for execution. If not provided, then
AIC device(s) would be auto-picked during execution. | List | **-** |
| `is_pre_quantized` | Boolean value to indicate if the model is pre-quantized. | Boolean | False |
| `cleanup` | Boolean value to denote if intermediate work\_dir artifacts should be
deleted during the flow. This is useful if there are space constraints
on user’s machine. | Boolean | False |
| `debug` | Boolean value to set debug log level | Boolean | False |
**Parent Topic:** [API](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-api.html)
## 11.6.1.2 Execution
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
QInference optimizer supports 3 modes of execution, and we have separate APIs for each
mode.
**Parent Topic:** [API](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-api.html)
## 11.6.1.2.1 Default mode
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
Executed using the quick\_run() API.
In quick mode, the tool attempts to evaluate and tune the model is minimum period of time
while providing a ballpark figure of the model’s performance. User can make decisions on
how to proceed accordingly. For example, proceed with exhaustive search using default
mode, make changes to the model, targets, etc.
Quick mode accepts only accuracy targets are arguments and is optional.
API params:
| Field Name | Description | Type | Default |
| --- | --- | --- | --- |
| `target_accuracy` | List of dictionaries of accuracy metric and target to achieve.
Example:
`{‘top1’: {‘target’: 0.99, ‘relative’: True},‘top5’:
{‘target’: 0.99,‘relative’: True}}` | Dictionary | **-** |
| `accuracy_lower_is_better` | Set if the metrics are better when their value is lower. | Boolean | False |
Code example:
result = qinf.run(target_accuracy={'top1': {'target': 0.99, 'relative': True},
'top5': {'target': 0.99,'relative': True}},
target_throughput=20000)
Copy to clipboard
**Parent Topic:** [Execution](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-execution.html)
## 11.6.1.2.2 Quick mode
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
Executed using the quick\_run() API.
In quick mode, the tool attempts to evaluate and tune the model is minimum period of time
while providing a ballpark figure of the model’s performance. User can make decisions on
how to proceed accordingly. For example, proceed with exhaustive search using default
mode, make changes to the model, targets, etc.
Quick mode accepts only accuracy targets are arguments and is optional.
API params:
| Field Name | Description | Type | Default |
| --- | --- | --- | --- |
| `target_accuracy` | List of dictionaries of accuracy metric and target to achieve.
Example:
`{‘top1’: {‘target’: 0.99, ‘relative’: True},‘top5’: {‘target’: 0.99,‘relative’: True}}`
| Dictionary | **-** |
| `accuracy_lower_is_better` | Set if the metrics are better when their value is lower. | Boolean | False |
Code example:
# Running quick mode of execution. "target_accuracy" is optional.
result = qinf.quick_run(target_accuracy={'top1': {'target': 0.99, 'relative': True},
'top5': {'target': 0.99,'relative': True}
})
Copy to clipboard
**Parent Topic:** [Execution](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-execution.html)
## 11.6.1.2.3 Expert mode
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
Executed using the expert\_run() API where user can provide the desired precision and
tuning type.
Here, the tool assumes that the user possesses adequate knowledge of the AIC backend and
is aware of the various options to use for model compilation, quantization, etc.
API params:
| Field Name | Description | Type | Default |
| --- | --- | --- | --- |
| `precision` | Option for user to select the precision for tuning the model. Allowed
options are `‘fp16’, ‘int8’, ‘mixed’`.
Note: Even when precision is chosen as mixed, Automatic Mixed
Precision will be attempted only if int8 quantization doesn’t meet
accuracy target. | String | fp16 |
| `tune` | Option for user to select if model tuning has to be done for
either accuracy only or performance only or both.
Allowed
options are `‘accuracy-only’, ‘performance-only’,
‘both’`. | String | both |
| `target_accuracy` | List of dictionaries of accuracy metric and target to achieve.
Example:
`{‘top1’: {‘target’: 0.99,
‘relative’: True},‘top5’:
{‘target’: 0.99,‘relative’:
True}}` | Dictionary | **Mandatory**, unless tune is set to ‘performance-only’ |
| `accuracy_lower_is_better` | Set if the metrics are better when their value is lower. | Boolean | False |
| `target_throughput` | Performance to be achieved in terms of throughput (inf/sec).
Performance tuning will be done for throughput if neither or both
target throughput and latency are provided. | Float | - |
| `target_latency` | Performance to be achieved in terms of latency (microseconds).
Recommended to use batchsize=1 to minimize latency. | Float | - |
| `performance_tuner_config` | Path to optimized search config file to be used with
model-configurator for performance tuning of the model. **If this option is not provided, then tool will estimate starting points from model analysis.** | String | - |
| `quantization_options` | Dictionary of quantization options for accuracy tuning, this is
applicable only when precision is ‘int8’ or ‘mixed’. Example:
`{ ‘quantization_calib’ : [‘Percentile’, ‘MSE’],
‘channelwise_quantization’ : [True, False],
‘percentile_values’ : [99.9, 99.99], ‘quantization_type’ :
[‘asymmetric’]}}` | Dictionary | **Mandatory when precision is ‘int8’ or ‘mixed’** |
Code examples:
Example 1: Below example demonstrates how to run only fp16 performance tuning.
This example shows a user provided a JSON file for performance tuning. This is optional
and when not provided, the tool would identify and generate this file based on model
analysis.
result_expert = qinf.expert_run(precision='fp16',
tune='performance-only',
performance_tuner_config='perf_config.json')
Copy to clipboard
Example 2: Below example demonstrates how to tune for accuracy only and with INT8
precision. In this example, since the provided accuracy targets are relative, reference
accuracy is calculated with FP32 precision and is used to calculate absolute accuracy
target scores.
result_expert = qinf.expert_run(
precision='int8',
tune='accuracy-only',
target_accuracy={'top1': {'target': 0.99, 'relative': True},
'top5': {'target': 0.99,'relative': True}},
quantization_options={
'channelwise_quantization': [True, False],
'quantization_calib': ['Percentile', 'SQNR'],
'percentile_values': [99.9, 99.99],
'quantization_type': ["asymmetric-symmetric", "asymmetric"]
})
Copy to clipboard
Example 3: Below example demonstrates how to run tuning for both accuracy and
performance in mixed precision. Even though accuracy targets provided below are absolute
values but still fp32 accuracy is calculated since it will be used to calculate allowed
drop accuracy for Auto Mixed Precision
result_expert = qinf.expert_run(
precision='mixed',
tune='both',
target_accuracy={'top1': {'target': 0.8, 'relative': False},
'top5': {'target': 0.9,'relative': False}},
target_throughput=20000,
quantization_options={
'quantization_calib': 'Percentile',
'percentile_values': 99.99,
'quantization_type': 'asymmetric-symmetric'
})
Copy to clipboard
**Parent Topic:** [Execution](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-execution.html)
## 11.6.2 CLI
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
The QInference Optimizer can also be run using CLI and this is done using the script
`/opt/qti-aic/tools/qaic-inference-optimizer/qinf_optimizer.py`
Arguments for CLI are below:
| Field Name | Description | Type | Default |
| --- | --- | --- | --- |
| `-config` | Configuration YAML file with common\_params, model\_preparation\_params
and accuracy\_evaluation\_params | String | **Mandatory** |
| `-model` | Path to ONNX model file | String | **Mandatory** |
| `-model-arch` | Architecture of the model, e.g., bert, yolov5 | String | unknown |
| `-relative-accuracy` | Relative accuracy targets, e.g., `‘{“f1”:
99}’` | JSON string | **Mandatory** |
| `-absolute-accuracy` | Relative accuracy targets, e.g., `‘{“f1”:
0.85}’` | JSON string | **Mandatory** |
| `-accuracy-lower-is-better` | Set if the metrics are better when their value is lower, e.g.,
perplexity | Boolean | False |
| `-target-throughput` | Performance to be achieved in terms of throughput (inf/sec) | Float | - |
| `-target-latency` | Performance to be achieved in terms of latency (microseconds).
Recommended to use batchsize=1 to minimize latency. | Float | - |
| `-work-dir` | Path to directory for storing intermediate artifacts and
results | String | qinf\_temp |
| `-quick` | Boolean to run quick mode. | Boolean | False |
| `-expert` | Boolean to run expert\_mode
Below expert mode params should be provided along with this
option. | Boolean | False |
| `-performance-tuner-config` | Path to optimized search config file to be used with
model-configurator for performance evaluation of the model | String | None |
| `-precision` | Option for user to select the precision for tuning the model. Allowed
options are `‘fp16’, ‘int8’, ‘mixed’`.
Note: Even when precision is chosen as mixed, Automatic Mixed
Precision will be attempted only if int8 quantization doesn’t meet
accuracy target. | String | fp16 |
| `-tune` | Option for user to select if model tuning has to be done for either
accuracy only or performance only or both.
Allowed options are
‘accuracy-only’, ‘performance-only’, ‘both’Copy to clipboard | String | both |
| `-quantization-calib` | Space separated quantization calibration options
(‘Percentile’ ‘MSE’)Copy to clipboard | String | - |
| `-channelwise-quantization` | Space separated channelwise quantization options
(‘True’ ‘False’)Copy to clipboard | String | - |
| `-percentile-values` | Space separated percentile values
(’99.9’ ’99.999’)Copy to clipboard | String | - |
| `-quantization-type` | Space separated quantization type
(‘asymmetric-symmetric’ ‘symmetric_with_uint8’)Copy to clipboard | String | - |
| `-aic` | Space separated AIC device IDs to use for execution. If not provided,
then AIC device(s) would be auto-picked during execution. | List | - |
| `-is-prequantized` | Boolean to indicate if the given model is pre-quantized | Boolean | False |
| `-clean` | Delete intermediate artifacts from “work-dir” | Boolean | False |
| `-debug` | Set debug log level | Boolean | False |
Below are few examples of CLI usage with each execution mode
Example 1: Default mode
$ python qinf_optimizer.py -config 'resnet50.yaml' \
-model "resnet50.onnx" \
-model-arch resnet50 \
-work-dir 'resnet50_cli_default' \
-aic 0 \
-relative-accuracy '{"top1":0.9,"top5":0.9}' \
-target-throughput 8000
Copy to clipboard
Example 2: Quick mode
$ python qinf_optimizer.py -config 'resnet50.yaml' \
-model "resnet50.onnx" \
-model-arch resnet50 \
-work-dir 'resnet50_cli_quick' \
-aic 0 \
-relative-accuracy '{"top1":0.9,"top5":0.9}' \
-target-throughput 8000 \ -quick
Copy to clipboard
Example 3: Expert mode
$ python qinf_optimizer.py -config 'resnet50.yaml' \
-model "resnet50.onnx" \
-model-arch resnet50 \
-work-dir 'resnet50_cli_expert' \
-aic 0 \
-relative-accuracy '{"top1":0.9,"top5":0.9}' \
-target-throughput 8000 \
-expert \
-precision mixed \
-tune both \
-quantization-calib Percentile \
-percentile-values 99.99 99.9 \
-quantization-type asymmetric-symmetric \
-channelwise-quantization False
Copy to clipboard
**Parent Topic:** [Usage](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qinference-optimizer-usage.html)
## 11.7 Limitations
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
Current limitations for QInference optimizer in this Release are:
- Only ONNX models are supported.
- Accuracy computations on AIC are always with batch size of 1.
- ONNX-runtime has a limitation with opset version of model to process QuantizeLinear
and Dequantizelinear operaters. Hence, Automatic Mixed Precision of the toolkit
works for models with opset version >= 10 and quantization is limited to layerwise
for models with opset version (10 to 12)
- For transformer-based models, user needs to mention the architecture as NLP.
- Large language models (LLMs) are not supported.
**Parent Topic:** [End-to-end workflow – QInference Optimizer](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
## 11.8 Appendix – Model Architecture
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
The tool optimizes tuning process based on model architecture. If aware, user can provide
model architecture from following available list. Else the tool will try to internally
analyze the model and determine the optimizations.
1. Classification
- RESNET18, RESNET50, RESNEXT101, MOBILENETV1, MOBILENETV2, VGG16, VGG19,
INCEPTIONV1, INCEPTIONV3, ALEXNET, DENSENET169, WIDERESNET50, BITM
2. Object Detection
- YOLOV3, YOLOV4, YOLOV5, RETINANET, CENTERNET, EFFICIENTDET, RESNET34\_SSD,
MOBILENETV2\_SSD
3. Segmentation
- DEEPLABV3PLUS, MASKRCNN\_R50\_FPN
4. Natural Language Processing (NLP)
- NLP, BERT, BERT\_BASE, DISTILBERT, DISTILGPT2, GPT3, ROBERTA\_BASE,
ROBERTA\_LARGE, DISTIL\_ROBERTA
**Parent Topic:** [End-to-end workflow – QInference Optimizer](https://docs.qualcomm.com/doc/80-PT790-993B/topic/end-to-end-workflow-qinference-optimizer.html)
Last Published: Jul 26, 2023
[Previous Topic
QAic device partition](https://docs.qualcomm.com/bundle/publicresource/80-PT790-993B/topics/network-deployment.md#network-deployment_qaic_device_partition) [Next Topic
Object detection postprocessing](https://docs.qualcomm.com/bundle/publicresource/80-PT790-993B/topics/Object-Detection-Post-processing.md)