# 8 Network performance tuning
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
**Parent Topic:** https://docs.qualcomm.com/doc/80-PT790-993B/topic/dl_inference_tools_part.html
## 8.1 Model configurator
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
An input model can be compiled with several network compilation and runtime parameters.
The model configurator tool helps identify the optimal compiler configuration for a network by running multiple configurations on the AIC100 device. The tool solves the following problems:
- How many cores to compile for? Using all the cores may not give the best performance for all models.
- What are the different compilation parameter values to choose?
- What batch size to use and how many instances to run in parallel?
The model configurator provides two ways to find the optimal compiler and runtime configuration for a model:
1. Brute force
The user can define a list of search space parameters (see [Search space parameters](https://docs.qualcomm.com/doc/80-PT790-993B/topic/model-configurator-search-space-parameters.html)). All valid combinations of the search space for a given model and inputs are evaluated. The performance achieved with each configuration is listed, so that the best combination of options can be determined.
2. Config optimizer
This method uses an optimization algorithm to conduct the search. For a given search space and initial value for the optimization parameters, the algorithm returns a locally optimal configuration. This is expected to arrive at the optimal solution faster, as it does not evaluate the entire search space, but instead tries to take informed variable-sized steps towards the optimal solution. [Search space parameters](https://docs.qualcomm.com/doc/80-PT790-993B/topic/model-configurator-search-space-parameters.html) describes how to use the config optimizer. Single device partitioning and depth first are not supported yet as search parameters in this method. Remote execution is also not supported.
The model configurator tool uses the qaic-exec tool for compiling the model and the qaic-runner tool for running the model.
The tool can be found in:
/opt/qti-aic/scripts/qaic-model-configuratorCopy to clipboard
A README for the tool is also available at ` /opt/qti-aic/scripts/qaic-model-configurator/README.md`.
Figure : Model configurator
**Parent Topic:** [Network performance tuning](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
## 8.1.1 Usage
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
model_configurator.py [-h] [-cores PARAM_CORES] [-batchsize PARAM_BS] [-mos PARAM_MOS] [-ols PARAM_OLS] [-dealloc-dly PARAM_DEALLOC_DELAY] [-split-size PARAM_SPLIT_SIZE] [-limit-vtcm-percent PARAM_VTCM_RATIO] [-enable-single-device-partitioning] [-test-depth-first-search] [-depth-first-mem PARAM_DFS_MEM] [-convert-to-fp16] [-quantization-precision {Int8,Int16}] [-quantization-precision-bias {Int8,Int32}] [-quantization-schema-activations {symmetric,asymmetric,symmetric_with_uint8,symmetric_with_power2_scale}] [-quantization-schema-constants {symmetric,asymmetric,symmetric_with_uint8,symmetric_with_power2_scale}] [-quantization-calibration {KLMinimization,Percentile,KLMinimizationV2,MSE,SQNR}] [-percentile-calibration-value PERCENTILE_CALIB] [-num-histogram-bins NUM_HISTOGRAM_BINS] [-quant-profile-threads QUANT_PROFILE_THREADS] [-quant-num-inputs QUANT_NUM_INPUTS] [-no-quant] [-external-quantization EXT_QUANT_FILE] [-node-precision-info NODE_PRECISION_FILE] [-execute-nodes-in-fp16 NODES_IN_FP16] [-keep-original-precision-for-nodes NODES_IN_ORIGINAL_PRECISION] [-device-processing] [-multicast-weights] [-combine-inputs {True,False}] [-combine-outputs {True,False}] [-compile-only] [-enable-depth-first] [-onnx-define-symbol-batch-size ONNX_DEFINE_SYMB_BS] [-onnx-define-symbol ONNX_SYMBS] [-stats-batchsize STATS_BATCHSIZE] [-max-compilation-threads MAX_COMPILATION_THREADS] [-iter ITER] [-time RUN_DURATION] [-instances INSTANCES] [-set-size SET_SIZE] [-input-list-file INPUT] [-input-list-generate] [-benchmark] [-custom-IO-list-file CUSTOM_IO_FILE] [-output-dir OUTPUT] [-device-id DEVICE_ID] [-model-input MODELINPUTDATA] [-output-node-name OUTNODENAME] [-qdrive] [-qdrive-telnet-ip QD_TELNET_IP] [-register-custom-op CUSTOM_OP_CONFIG] [-enable-channelwise] [-enable-rowwise] [-image-dir IMAGE_DIR] [-image-type {jpg,png}] [-height IMGHEIGHT] [-width IMGWIDTH] [-letterbox-resize] [-no-nhwc-conversion] [-no-normalize] [-normalize-255] [-fp32-to-int8] [-bgr-to-rgb] [-num-raw-files RAW_FILES_CNT] [-reuse-single-file] [--verbose] [-save-configs SAVESPFILE] [-load-configs LOADSPFILE] [-list-configs] [-remote-exec REMOTE_EXEC_INFO] [-load-pre-gen-files PRE_GEN_FILES] [-create-batched-inputs CREATE_BATCHED_INPUTS] [-parallel-evaluator] [-num-parallel-compile NUM_PARALLEL_COMPILATIONS] [-retain-binaries] [-optimized-config-search OPT_CONFIG_SEARCH]
MODEL {onnx,tf,caffe,caffe2,pytorch} Copy to clipboard
Details of each of option can be found by running:
python model_configurator.py –help Copy to clipboard
**Dependencies:** Python 3.8, Python packages - opencv-python, pandas, numpy, sklearn dataclasses, paramiko, jsonschema
The dependent packages can be installed with:
pip3 install -r /opt/qti-aic/scripts/qaic-model-configurator/requirements.txtCopy to clipboard
The model configurator requires both the Platform SDK and the Apps SDK to be installed.
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
## 8.1.2 Supported model types
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
Table : Supported model types
| No. | Type | Command syntax |
| --- | --- | --- |
| 1 | Onnx | python model\_configurator.py **<path to .onnx model> onnx** <other options> |
| 2 | Caffe | python model\_configurator.py **<path to directory containing caffe model> caffe**<other options> |
| 3 | Caffe2 | python model\_configurator.py **<path to directory containing caffe2 model>** **caffe2 -model-input**=*<inputName1>,<inputType1>,<inputShape1>* <other options>
**Note:** When running with multiple batch sizes, use an **asterisk** in place of first dimension for model-input.
Example: `-model-input=gpu_0/data,float,[*,3,224,224] ` |
| 4 | Tensorflow | python model\_configurator.py **<path to tf model pb> tf -output-node-name**=<comma separated names of outputs> <other options> |
| 5 | PyTorch | python model\_configurator.py **<path to PyTorch model pt> pytorch -model-input=**<inputName1>,<inputType1>,<inputShape1> <other options>
**Note**: When running with multiple batch sizes, use an asterisk in place of the first dimension for model-input.
Example: `-model-input=gpu_0/data,float,[*,3,224,224] ` |
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
## 8.1.3 Search space parameters
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
## 8.1.3.1 Brute force
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
The search space for brute force can be created by using the options in the following table. All the parameters have default values that can be found by using `-help` on `model-configurator.py`.
Table : Search Space parameters
| No. | Parameter | Command syntax |
| --- | --- | --- |
| 1 | `-cores` | Comma separated values and ranges (Examples: 1-4,8,12).
Number of cores – Selects the number of cores that the model will be compiled for. This option instructs the AIC100 compiler on how many cores the model compilation should be distributed. The number of cores should not exceed the number of cores available on the hardware.
To know the number of cores available on the hardware, use the command "`/opt/qti-aic/tools/qaic_util -q`" (available once the Platform SDK is installed). |
| 2 | `-batchsize ` | Comma separated values and ranges (Examples: 1-4,8,12).
It indicates the batch size of the model input.
Multiple inputs can be batched and inferenced together to reduce control and data path overhead. |
| 3 | `-mos` | Maximum output channel split – The effort level to reduce the on-chip memory usage. The compiler optimizes for the on-chip memory usage by mapping the network to the on-chip memory.
Increasing the effort level holds more of the network inside on-chip memory, which may lead to a higher communication overload.
There may be a sweet spot for optimum performance that is dependent on the actual network being run.
This value should be less than or equal to the number of cores.
The list of MOS values to be searched across as set by the user is augmented with an additional default value. For points in the search space where the MOS equals this default value, the compiler uses its own internal heuristic algorithms to determine a MOS value to use.
If this option is not set, then the compiler sets it as per its internal heuristic algorithms. |
| 4 | `-ols` | Overlap split factor – Factor to increase splitting of network operations.
The compiler can split network nodes into multiple instances to enable more fine-grained parallelism. Generally higher is better, but there may be a sweet spot that may need to be tuned for a given network.
This value should be less than or equal to 8.
If this option is not set, then the compiler sets it as per its internal heuristic algorithms. |
| 5 | `-instances ` | Comma separated values and ranges (Examples: 1-4,8,12).
It indicates the concurrent thread number to execute the same model in parallel.
By default, instances \* cores <= the number of cores available on the hardware. |
| 6 | `-depth-first-mem` | Comma separated values and ranges. It indicates the memory size for depth first compiler optimizations. Works when depth-first is enabled and for cores=1. To enable depth-first, use option `-enable-depth-first`. See [Other options](https://docs.qualcomm.com/doc/80-PT790-993B/topic/model-configurator-other-options.html).
If only `-enable-depth-first` is specified, then the compiler chooses the depth-first-mem value using some heuristics. |
| 7 | `-dealloc-dly` | Comma separated and ranges. Example: 0-2,5.
Sets buffer lifetime. Adjusts how long to keep allocation alive past dealloc.
Smaller values may result in higher performance for networks with large inputs.
Valid values are in range [0,10]. Steps of 1 are taken for the specified range. |
| 8 | `-split-size` | Comma separated and ranges. Example: 512-1024,2048.
Sets the maximum tile size in KiB. Valid values are in range [512,2048].
Steps of 256 are taken for the specified range. |
| 9 | `-limit-vtcm-percent` | Comma separated and ranges. Example: 50,80,100.
Percentage of fast memory an instruction can use. Valid values are in range [0,100]. Steps of 5 are taken for the specified range. |
**Parent Topic:** [Search space parameters](https://docs.qualcomm.com/doc/80-PT790-993B/topic/model-configurator-search-space-parameters.html)
## 8.1.3.2 Config optimizer
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
To run the optimized search, provide the search arguments via a JSON configuration file. The optimized search can be run on one or more of the searchable parameters (cores, mos ols, batch-size, dealloc-delay, split-size, limit-vtcm-percent and instances). Static values can be set for searchable parameters that have been excluded from the search space.
Table : Config optimizer JSON configuration file
| Parameter | Command syntax |
| --- | --- |
| `-optimized-config-search ` | Path to the configuration file. |
Sample configuration files are available at:
/opt/qti-aic/scripts/qaic-model-configurator/SampleFiles/ optimizer_config_full_search.json
/opt/qti-aic/scripts/qaic-model-configurator/SampleFiles/ optimizer_config_limited_search.json
/opt/qti-aic/scripts/qaic-model-configurator/SampleFiles/ optimizer_config_limited_search_static_bs_ols.jsonCopy to clipboard
The JSON file requires the elements listed in the following table.
Table : Config optimizer JSON file elements
| No. | Key | Type | Description | Recommended Value |
| --- | --- | --- | --- | --- |
| 1 | "max\_func\_eval" | Integer | Maximum number of evaluations to do for each initial point. This number can be increased if successful convergence is not achieved. | 200 |
| 2 | "objective" | String | Search objective. Options "maximize\_inf\_rate" or "minimize\_latency". | |
| 3 | "params" | JSON Object | Provide the search range for each of the parameters – cores, mos, ols, and so on through the min, max values. | Set full range of valid values for each of the options.
Refer to the “Parameter Range” table below. |
| 4 | "static\_params" | JSON Object | Optional static values to be used for searchable parameters that have been excluded from the search space. | |
| 5 | "initial\_values" | List of JSON Objects | List of initial values for the search parameters. A fresh search is initiated from each of these points and the results returned. Initial values must be picked from within the search range defined in "params". | Provide multiple initial values.
Refer to the “Guideline for setting initial value” below. |
The recommended values are meant to be a general guidance and starting point. They may not always apply. For example, some models might be too big to execute for a larger batch size, in which case the range should be chosen accordingly. The following table contains the recommended search range to be set.
Table : Parameter range
| No. | Parameter | Min | Max | Comment |
| --- | --- | --- | --- | --- |
| 1 | cores | 1 | Number of NSPs on device | |
| 2 | mos | 1 | Number of NSPs on device | |
| 3 | ols | 1 | 8 | Optimal value is typically found in range 1-8.
However, the ols value can typically be any integer > 1. |
| 4 | batch-size | 1 | 64 | The Min and Max values must be a power of 2.
The Max value would depend on the model. |
| 5 | instances | 1 | Number of NSPs on device | |
| 6 | dealloc-dly | 0 | 4 | Valid range: [0-10]
Most models have an optimal value between 0-4. |
| 7 | split-size | 512 | 2048 | |
| 8 | limit-vtcm-percent | 25 | 100 | Valid range: [0,100]
Most models have an optimal value between 25-100. |
## Setting initial values
Initial values are critical to how the search is conducted and how fast the algorithm will converge. If the user has general intuition on the optimal solution, then the initial value can be picked based on this.
It is recommended to set around 4-5 initial values configured using one of the options below:
1. Sweep over the cores in fixed-sized steps, set vtcm to 100, and keep the initial values for the other parameters to their Min value. For example, cores can be set to [1,4,8,12,16] taking steps of 4 for a 16 NSP device.
2. Sweep over the cores in fixed-sized steps as in Option **1**. Choose the corresponding instances value, such that `cores * instances` is close to number of NSPs on the device.
Example: Pick `{cores,instances}` pairs from `[{1,16}, {4,4}, {8,2}, {12,1}, {16,1}]` for a 16 NSP device. Set vtcm to 100 and the rest of the parameters to their Min value.
With config optimizer, the model is always run using benchmark mode. The batch mode for running models through config optimizer is not supported yet. For more details on the benchmark/batch mode, refer to the **benchmark** option.
**Parent Topic:** [Search space parameters](https://docs.qualcomm.com/doc/80-PT790-993B/topic/model-configurator-search-space-parameters.html)
## 8.1.4 Optimized search results
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
The results of the optimized config search will include:
- The initial value used for that run of the optimizer.
- The status returned by the optimizer (the following table lists the possible statuses).
- The configuration to which the optimizer converged.
- The objective value that was achieved.
The optimizer status should be checked before interpreting the rest of the results.
Table : Optimizer status values
| No. | Optimizer status | Comment |
| --- | --- | --- |
| 1 | SUCCESS | The optimizer successfully converged on a solution. |
| 2 | CONSTRAIN\_FAILURE | The optimizer stopped without meeting all the constraints. |
| 3 | NUMERICAL\_FAILURE | The optimizer encountered a numerical failure internally. |
| 4 | EXCEED\_MAX\_ITER | The optimizer exceeded the max allowed number of iterations of its algorithm. Max\_iterations = 2 \* max\_evaluations. |
| 5 | EXCEED\_MAX\_EVAL | The optimizer exceeded the max allowed number of evaluations. |
| 6 | UNKNOWN\_ERROR | The optimizer returned an unknown error. |
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
## 8.1.5 Input file options
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
Table : Input file options
| No. | Parameter | Command syntax |
| --- | --- | --- |
| 1 | -input-list-file | Provide an input text file with list of raw input file paths.
**Use this option when you already have a raw file with the correct batch size as expected by the model.**
**With this option you can run with only one batch size.** |
| 2 | -input-list-generate | Generates raw files on the fly for different batch sizes from JPG/PNG images in a directory.
**Use this option when you have a directory of PNG/JPG images, and you want to run across different batch sizes.**
The following parameters are mandatory to be passed if this option is chosen:
`-image-dir` <Directory of all images - should contain at least number of max batch size images`>`
If there are lesser files that the required batch size, user can pass:
`-reuse-single-file` option to reuse the same file for multiple batches.
`-height` < height of the image the model takes>
`-width` <width of the image the model takes>
`-image-type` <extension of images in image-dir. Default jpg>
For other optional parameters with this input type, refer to the "`Input file Generation Options`" section in `--help`. |
| 3 | -load-pre-gen-files | Pregenerated input data files and PGQ profiles for a batch size can be provided through this option. This is an extension for `-input-list-file` which is for one batch size alone. File paths should be absolute paths or relative paths to parent directory of JSON file.
For specification, an example file can be found at: `/opt/qti-aic/scripts/qaic-model-configurator/SampleFiles/pre_gen_files.json ` |
| 4 | -create-batched-inputs | Text file with comma-separated ordered input .raw files with no batch dimension. Add lines for a new set of inputs. The inputs will be concatenated together to create a batch on-the-go. File paths of the .raw files should be absolute paths or relative paths to parent directory of the txt file. Assumes all inputs are Int64.
Sample file at: `/opt/qti-aic/scripts/qaic-model-configurator/SampleFiles/create_batched_inputs_file.txt ` |
| 5 | None of the above | Generates random numbers (default Gaussian) for the required size of inputs. This option may perform slower than providing real input. |
Note: If the inputs provided are less than the –iter specified, a warning is issued. The inputs are repeated to meet the –iter numbers. If the inputs provided are greater than the –iter specified, then all the inputs are executed, so the actual –iter run will be greater than the –iter specified.
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
## 8.1.6 Quantization options
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
Table : Quantization options
| Parameter | Command syntax |
| --- | --- |
| `-quantization-precision ` | Quantization precision: Int8 (default) or Int16 |
| `-quantization-precision-bias ` | Quantization precision bias: Int8 (default) or Int32 |
| `-quantization-schema-activations` | Quantization scheme to use for activations:
symmetric, symmetric\_with\_uint8 (default), symmetric\_with\_power2\_scale |
| `-quantization-schema-constants` | Quantization scheme for to use for constants such as weights and bias:
symmetric, symmetric\_with\_uint8 (default), symmetric\_with\_power2\_scale |
| `-quantization-calibration` | Specify which quantization calibration to use: KLMinimization, Percentile, MSE, SQNR, or KLMinimizationV2.
If not specified, default (None – MinMax calibration) is used. |
| `-percentile-calibration-value` | Specify the value to be used with the percentile calibration method.
The specified float value must be within 90 and 100. Default is 100. |
| `-num-histogram-bins ` | Sets the number of histogram bins that will be used in profiling every node. Default is 512. |
| `-convert-to-fp16 ` | Run all floating-point computation in fp16 (disabled by default).
No quantization is performed. |
| `-no-quant ` | Disables quantization. |
| `-node-precision-info` | Load node precision config file to run specific instances in FP16. |
| `-execute-nodes-in-fp16` | Run all instances of the operators in this list with FP16. |
| `-keep-original-precision-for-nodes ` | Run all instances of the operators in this list with original precision. |
| `-external-quantization` | Provide the external quantization file path. |
| `-custom-IO-list-file` | Custom I/O configuration file. |
| `-load-pre-gen-files` | Pregenerated input data files and PGQ profiles for a batch size can be provided through this option. This is an extension for `-input-list-file`, which is for one batch size alone. File paths should be absolute paths or relative paths to parent directory of the .json file.
For specification, an example file can be found at:
`/opt/qti-aic/scripts/qaic-model-configurator/SampleFiles/pre_gen_files.json`. |
| `-enable-rowwise` | Enable row-wise quantization. |
| `-enable-channelwise` | Enable channel-wise quantization. |
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
## 8.1.7 Other options
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
Table : Other options
| Parameter | Command syntax |
| --- | --- |
| `-load-configs, `
`-save-config, `
`-list-configs ` | Options used to save the search space in a file and load it later for another run (with editing if needed).
The `-list-configs` option will list all the executions before running. |
| `-iter ` | Number of iterations to run. |
| `-device-processing ` | Enable pre- and postprocessing on the AIC device. |
| `-compile-only ` | Perform compilation only (no execution).
Compiled networks are saved in output directories. |
| `-device-id ` | AIC device ID in case of multiple devices. |
| `-d32-spatial-major ` | Use spatial major layout. Note that the compiler automatically uses spatial major layout for depth-wise separable convolutions, so it is not necessary to specify this option explicitly for such cases. |
| `-qdrive ` | Use qdrive present at 192.168.1.1 instead of attached SA9000 to execute each model. |
| `-register-custom-op ` | Register custom operations for your model using the configuration file. |
| `-enable-depth-first ` | Enables depth-first optimizations. Compiler sets the default depth-first memory size. To search across different depth-first memory sizes use the option
`-depth-first-mem=N.` See *Search space parameters* for more information. |
| `-enable-single-device-partitioning` | Enables single-device partitioning. The cores involved are split into partitions/clusters and the network is split among the clusters, allowing for pipelined execution of the network.
Extends the search space by automatically generating valid cluster configurations and the associated MOS values for each cluster. |
| `-remote-exec ` | Enables remote execution of the model. Pass host and username as username@host. The connection is geared to be password-less and requires the ssh keys to be set up on the remote machine for this to work. When specified, the model is compiled locally, the compiled binary and input data are copied to remote machine, and model is run there. This is expected to be slower as data movement is involved. |
| `-onnx-define-symbol-batch-size ` | Set the onnx symbol name for batch size. Sets the batch size through the`onnx-define-symbol` option.
The batch size values are picked from the `-batchsize` option. This option is relevant for ONNX models alone. |
| `-retain-binaries ` | By default, compiled model binaries are deleted as they can take up a lot of disk space.
This option can be used to retain the binaries. |
| `-max-compilation-threads` | Max number of threads to spawn for running multiple compilations in parallel. |
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
## 8.1.8 Debug issues
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
When running the configurator over a search space, a set of files are created in the output directory. If no output directory is specified by the user, it defaults to directory name “model\_configurator\_output/” in the directory from which the command is being run.
Table : Model configurator logs
| Log files/directory | Description |
| --- | --- |
| model\_configurator.log | Main application log file. |
| results.csv | The final results are saved in csv. |
| compiled\_models | Directory containing compiler outputs. Removed at application exit, unless the`-retain-binaries` command line option is specified. |
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
## 8.1.9 Example config files
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
Example config files (for use with model\_configurator) are included in the `SampleConfigurations` sub-directory.
The file `config_8cores_only_smallset.csv` can be used with model\_configurator for an 8-core AIC100 SoC to do a shallow search of parameters values. This config file restricts the search space to combinations that satisfy the following conditions: cores\*instances = 8, batchsize=1 or 4, mos=1,4, or 8, ols=1,2, or 4. It has 36 parameter combinations and may be helpful to quickly gauge the parameters to which a given network is sensitive and hence worth exploring with a deeper search.
The file `config_8cores_only.csv` is also suited for use with an 8-core AIC100 SoC; however, it specifies a deeper search over the same parameters as `config_8cores_only_smallset.csv`. It also targets configurations that satisfy the criterion cores\*instances = 8; however, it sweeps a larger range of values for the other parameters. Specifically, batchsize=1,2,4, or 8, mos=1,2,4, or 8, and ols = 1,2, or 4. The file has 104 combinations and hence model\_configurator will take a longer time to finish.
It is recommended that model\_configurator be invoked with one of the above config files for new networks to determine if there is an opportunity to tune parameters further. The default parameter values may not always be optimal.
The file `config_14cores_ref.csv` is provided as a reference that contains all valid combinations of cores and instances for a 14-core AIC100 SoC. It sweeps over batchsize=1,2,4,8, or 14, mos=1,2,4, 8, or 14 and ols=1,2, or 4. It has 304 combinations, including some that use fewer than 14 cores for the combination of cores and instances. The following command is one example that can generate this configuration file:
$ python3 /opt/qti-aic/scripts/qaic-model-configurator/model_configurator.py vgg16.onnx onnx -list-configs -save-configs ./config_14cores_ref.csv -iter 500 -cores=1,2,4,8,14 -instances 1,2,4,8,14 -batchsize 1,2,4,8,14 -mos 1,2,4,8,14 -ols 1,2,4Copy to clipboard
By filtering out just the combinations of interest, other config files can be derived from this reference file. For example, `config_8cores_only_smallset.csv` can be derived from `config_14cores_ref.csv` by selecting only combinations for which cores\*instances=8, batchsize=1 or 4, mos=1,4, or 8, ols=1,2, or 4.
The selected config file can be passed to `qaic-model-configurator` to execute a search using the `-load-configs` option. For example:
$ python3 /opt/qti-aic/scripts/qaic-model-configurator/model_configurator.py vgg16.onnx onnx -output-dir int8host -list-configs -load-configs ./config_14cores_ref.csv -iter=500 -quantization-schema-activations=symmetric_with_uint8 -quantization-precision=Int8 -input-list-generate -image-dir ./ -height 640 -width 640 -image-type jpg -reuse-single-fileCopy to clipboard
Note: Options that impact how a model should be executed, such as quantization and format conversion options should be added to a `qaic-model-configurator` invocation that performs the actual measurements, such as the one above.
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
## 8.1.10 Limitations
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
Running on CentOS-7:
- To run qaic-model-configurator on a CentOS-7 host, the libstdc++ library that is
packaged in the Apps SDK must be used. This requires setting the environment
variable LD\_LIBRARY\_PATH before running qaic-model-configurator. This can be done
with the following command:
export LD_LIBRARY_PATH=/opt/qti-aic/dev/lib/x86_64:/opt/qti-aic/dev/lib/x86_64/appsCopy to clipboard
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
## 8.1.11 Example run and img2raw usage for a ResNet-50 model
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/network-perform-tuning.html)
For Caffe2 models:
$ python3 /opt/qti-aic/scripts/qaic-model-configurator/model_configurator.py caffe2_resnet50_caffe2 -iter 100 -list-configs -batchsize 1,4,8 -cores 2,8 -mos 2 -ols 2 -instance 1 -input-list-generate -image-dir $PATH_IMAGE_SET -width 224 -height 224 -model-input=gpu_0/data,float,[*,3,224,224] -reuse-single-fileCopy to clipboard
For ONNX models:
$ python3 /opt/qti-aic/scripts/qaic-model-configurator/model_configurator.py resnet50.onnx onnx -iter 5000 -precision Int8 -list-configs -batchsize 1 -cores 2,7-12 -mos 2 -ols 2 -instance 2,7 -input-list-file inputs_res.txtCopy to clipboard
For an ONNX model with single device partitioning:
$ python3 /opt/qti-aic/scripts/qaic-model-configurator/model_configurator.py SsdResnet18_1MP.onnx onnx -iter 5000 -list-configs -batchsize 1 -cores 6 -mos 1,2,4,8 -ols 2,4 -instance 1 -input-list-file list.txt -width 224 -height 224 -reuse-single-file -enable-single-device-partitioningCopy to clipboard
**To generate raw files** from jpg/png images for different batch sizes, the `img2raw` Python script can be used. This script is imported by the model\_configurator to generate raw files for different batch sizes from a provided image directory.
img2raw.py [-h] [-batchsize BATCH_SIZE] -image-dir IMAGE_DIR
[-image-type {jpg,png}] -height IMGHEIGHT -width IMGWIDTH
[-no-nhwc-conversion] [-no-normalize] [-fp32-to-int8]
[-bgr-to-rgb] [-num-raw-files RAW_FILES_CNT]
[-reuse-single-file] [-output OUTPUT]
-h, --help show this help message and exit
-batchsize BATCH_SIZE Output batch size, default 1
-image-dir IMAGE_DIR Input image directory
-image-type {jpg,png} jpg (default) or png
-height IMGHEIGHT output image height
-width IMGWIDTH output image width
-no-nhwc-conversion Disable conversion from NHWC to NCHW
-no-normalize Disable Normalization
-fp32-to-int8 Convert input from Fp32 to int8
-bgr-to-rgb Convert from BGR to RGB
-num-raw-files RAW_FILES_CNT Number of raw files to generate default 1 (multiple of batchsize)
-reuse-single-file If set will reuse same image for multiple batches
-output OUTPUT Output Directory Copy to clipboard
**Example img2raw.py**
(Picks up four images from JPEGImages\_dir, does scaling with provided width and height, NHWC→NCHW, normalization, and merges the buffers into a single raw image.)
$ python3 /opt/qti-aic/scripts/qaic-model-configurator/img2raw.py -image-dir JPEGImages_dir -height 124 -width 124 -batchsize 4
ls batch_size_4
file-list.txt img_0.rawCopy to clipboard
Table : img2raw options
| No. | Option | Description |
| --- | --- | --- |
| 1 | `-batchsize` | Output batch size, default 1. |
| 2 | `-image-dir IMAGE_DIR` | Input image directory. There could be any number of files in the image-dir (should be more than the batch size \* num-raw-files). |
| 3 | `-image-type Type` | Type of images in the image-dir (png/jpg). |
| 4 | `-height IMGHEIGHT`,
`-width IMGWIDTH` | Output image dimensions. Input images are scaled to these dimensions. |
| 5 | `-no-nhwc-conversion` | Input images are expected in NHWC format and converted to NCHW. Add this flag to disable this conversion. |
| 6 | `-no-normalize` | Disable normalization of images. |
| 7 | `-fp32-to-int8` | Convert the output images in int8 format. (Disabled by default.) |
| 8 | `-bgr-to-rgb` | Convert input images from BGR to RGB format. (Disabled by default.) |
| 9 | `-num-raw-files` | How many raw files to generate (Default 1).
Example: If -batchsize=8 and if 16 images are present in the directory, two raw files will be created.
The image-dir should have at least batch size \* num-raw-files files. |
| 10 | `-reuse-single-file` | If set, will reuse the same image for generating multibatch raw files. Only a single image from the image-dir will be used. |
| 11 | -output | Output directory where the files will be created. A text file is also created in the same directory with the names of the raw files generated. |
| 12 | `-letterbox-resize` | Resize image as per letterbox logic. |
| 13 | `-normalize-255` | Normalization by dividing image by 255. |
| 14 | `-iter` | Number of iterations (iter) for which input file should be generated. |
**Parent Topic:** [Model configurator](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-model-configurator.html)
Last Published: Jul 26, 2023
[Previous Topic
Remote execution over ssh with preshared key setup](https://docs.qualcomm.com/bundle/publicresource/80-PT790-993B/topics/network-accuracy-analysis.md#network-accuracy-analysis_remote_execution_configuration) [Next Topic
Network profiler](https://docs.qualcomm.com/bundle/publicresource/80-PT790-993B/topics/network-profiler.md)