# 12 Object detection postprocessing
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
**Parent Topic:** https://docs.qualcomm.com/doc/80-PT790-993B/topic/dl_inference_tools_part.html
## 12.1 QAic Smart NMS
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
Smart NMS (non-max suppression) provides a way to run parts of the network on AI accelerators and other parts, on which better overall inference times can be achieved by leveraging parallelism across two devices. The object detection models can be partitioned to run the feature extractor part on the AI100, and to run the remaining box processing and NMS modules on the host.
**Parent Topic:** [Object detection postprocessing](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
## 12.1.1 Installation
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
Smart NMS is provided via two interfaces:
- Pre compiled executable (qaic-smart-nms)
- Precompiled library (libAICsmartnms.so)
- Python wheel package (smartnms-x.x.x-cp38-cp38-linux\_x86\_64.whl)
The “`qaic-smart-nms`” executable is part of the SDK release and can be found at “`/opt/qti-aic/tools/smart-nms/qaic-smart-nms`”. The library is present at “`/opt/qti-aic/tools/smart-nms/libAICsmartnms.so`”. The header files required with usage of the library are present at “`/opt/qti-aic/tools/smart-nms/src/include`”. The Python package is present at “`/opt/qti-aic/tools/smart-nms/smartnms-*.whl`”.
**Parent Topic:** [QAic Smart NMS](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-introduction.html)
## 12.1.2 Usage
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
**Parent Topic:** [QAic Smart NMS](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-introduction.html)
## 12.1.2.1 qaic-smart-nms
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
The executable tool can be used on the output of partitioned object detection models with the help of the configuration file specified for that model. The “`qaic-smart-nms`” tool requires model outputs, network-descriptor, and model-information as inputs. The information needs to be provided via a YAML configuration file to “`qaic-smart-nms`”. It outputs boxes, labels, scores, and counts in .bin files as outputs inside the directory specified in “`abp-output-directory`” inside the user-config file.
Example usage for the resnet34-ssd model:
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/qti-aic/tools/smart-nms/
$ cd /opt/qti-aic/tools/smart-nms
$ ./qaic-smart-nms --config ./src/examples/configs/user-config-resnet34ssd.ymlCopy to clipboard
The tool also outputs “`infer-output.txt`” inside the same output directory. The content of infer-output.txt will be a comma path to each output.
Example:
./ smart_nms_output/boxes-activation-0-inf-1.bin, ./smart_nms_output/scores-activation-0-inf-1.bin, ./ smart_nms_output/labels-activation-0-inf-1.bin, ./ smart_nms_output/counts-activation-0-inf-1.binCopy to clipboard
Table : qaic-smart-nms argument details
| Argument | Description |
| --- | --- |
| -c,--config DESTINATION | Specify the destination path to the configuration file. |
| -h,--help | Show the help message. |
Table : qaic-smart-nms (output Interpretations)
| Output file | Underlying datatype | Remarks |
| --- | --- | --- |
| `boxes-activation-0-inf-1.bin` | raw / float32 | This file contains all box\_coordinates[x,y,x,y] for all batches in flat format.
Example: If yolov3 is executed with batch-size = 4, (that is, four images), and in each image, there are 4, 10, 2, 21 detections respectively that are run under a single thread, then `boxes-activation-0-inf-{index}.bin` will have (4+10+2+21) \* 4, which is 148 float32 values. |
| `scores-activation-0-inf-1.bin` | raw / float32 | This file contains all scores for all batches in flat format.
Example: If yolov3 is executed with batch-size = 4, (that is, four images), and in each image, there are 4, 10, 2, 21 detections respectively that are run under a single thread, then `scores-activation-0-inf-{index}.bin` will have (4+10+2+21), which is 37 float32 values. |
| `labels-activation-0-inf-1.bin` | raw / float32 | This file contains class-idx for every box in boxes-activation-0-inf-{index}.bin in flat format.
Example: If yolov3 is executed with batch-size = 4, (that is, four images), and in each image, there are 4, 10, 2, 21 detections respectively, that are run under a single thread, then `labels-activation-0-inf-{index}.bin` will have (4+10+2+21), which is 37 float32 values. |
| `counts-activation-0-inf-1.bin` | raw / float32 | This file contains counts of detection for each batch format.
Example: If yolov3 is executed with batch-size = 4, (that is, four images), and in each image, there are 4, 10, 2, 21 detections respectively that are run under a single thread, then `counts-activation-0-inf-{index}.bin` will have four float32 values: 4,10,2,21. |
**Parent Topic:** [Usage](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-usage.html)
## 12.1.2.2 libAICsmartnms.so
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
libAICsmartnms.so can be linked with CPP code and the API mentioned in include files can be used in client CPP code. The example shows use of the APIs present in libAICsmartnms.so along with CMake to link to the library. The library requires GCC >= 7.5 , Protobuf==3.11.4 and CMake >= 3.15.
/*
Code for fp32 post-processing on resnet34-ssd model
*/
// include the required header files
// include directory to be specified in cmake
#include "AnchorBoxProcessing.h"
#include "Interface.h"
// Initiate the parameters required for smart box processing and NMS
std::string configfilepath = "./src/examples/configs/user-config-resnet34ssd.yml";
InitParameters params(configfilepath);
// Initiate the AnchorBoxProcessing class
// Available options for different precision - anchor::fTensor, anchor::uTensor, anchor::iTensor, anchor::hfTensor
AnchorBoxProcessing anchorBoxProc(params);
// Place holder for inputs to smart nms
std::vector detections1;
std::vector logits1;
std::vector lm1;
//Placeholder for outputs from smartbp-nms
// Output will store each vector of float
/*
[0] -> threadid
[1] -> box_coordinate_1
[2] -> box_coordinate_2
[3] -> box_coordinate_3
[4] -> box_coordinate_4
[5] -> box_score
[6] -> box_class_label
[7] -> batch_identifier
*/
std::vector> results(0,std::vector(7,0));
std::vector>> landmarksResults(0 , std::vector>(params.num_landmarks, std::vector (2, 0)));
// call the anchor box processing handle api
(anchorBoxProc.*(anchorBoxProc.handle))(detections1, logits1, lm1, results, landmarksResults, threadid);Copy to clipboard
The following changes in CMake are required to use the API(s) of Smart-NMS.
target_link_libraries(example PRIVATE /opt/qti-aic/tools/smart-nms/libAICsmartnms.so)
target_include_directories(example PRIVATE /opt/qti-aic/tools/smart-nms/src/include)Copy to clipboard
Table : libAICsmartnms.so (output vector interpretations)
| Output vector index | Interpretation | Remarks |
| --- | --- | --- |
| 0 | id | This is a placeholder for thread\_id. In the case of multithreaded execution, this can be used as book-keeping information. |
| 1,2,3,4 | box\_coordinates | These are the four coordinates of boxes that can be used to do mAP evaluation or plotting; these boxes need postprocessing based on model architecture. |
| 5 | box\_score | The box score for the class mentioned in box\_class\_label,
That is, a 0.9 as the box\_score and a box\_class\_label = 5, means that a box (with the box\_coordinates above) has a 90% chance to be class 5. |
| 6 | box\_class\_label | The class ID of the box. |
| 7 | batch\_identifier | If the model support batch-size >1, run under a single thread, then each row in outputs needs to be mapped to its correct batch. This information is captured in batch\_identifier.
Example: If yolov3 is executed with batch-size = 4, (that is, four images), and in each image there are 4, 10, 2, 21 detections that are run under a single thread, then the first 4 rows in output vector have batch\_identifier as 0, the next 10 will have 1, the next 2 will have 3, and finally the next 21 will have 3. |
The following changes in CMake are required to use the API(s) of Smart-NMS.
target_link_libraries(example PRIVATE /opt/qti-aic/tools/smart-nms/libAICsmartnms.so)
target_include_directories(example PRIVATE /opt/qti-aic/tools/smart-nms/src/include) Copy to clipboard
Example of a typical CMake that is required to run client application using libAICsmartnms.so:
cmake_minimum_required(VERSION 3.15)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_FLAGS -O3)
set(CMAKE_CXX_FLAGS "-std=c++17 -pthread -O3 -ffast-math -ftree-vectorize")
set(AIC_INCLUDE_DIR "/opt/qti-aic/dev/inc" "/opt/qti-aic/dev/inc/qaic-api-common")
list(APPEND PUBLIC_HEADERS
include
${AIC_INCLUDE_DIR}
)
list(APPEND SRCS
Client.cpp
)
add_executable(example
${SRCS})
set(AIC_LIBRARY ${CMAKE_CURRENT_BINARY_DIR}/libAICsmartnms.so)
set_target_properties(example PROPERTIES LINK_FLAGS -Wl,--unresolved-symbols=ignore-in-shared-libs)
target_link_libraries(example PRIVATE ${AIC_LIBRARY} ${CMAKE_DL_LIBS})
target_include_directories(example PRIVATE ${PUBLIC_HEADERS})Copy to clipboard
Users can now write a function for decoding boxes and pass that function object to the anchorBoxProcessing object.
The AnchorBoxProcessing class constructor accepts the InitParameters object with logic for decoding boxes and landmarks from prior information.
Box decoding is a std::function object with a type of
std::function&, const float *, const uint32_t&)>Copy to clipboard
and Landmark decoding is also std::function object with type of
std::function>&, const float *)Copy to clipboard
boxFunctor,lmFunctor will look for default decoding functions. If the default decoding functions do not match with the current model architecture, then it will issue an error.
Refer to the following sample for initialization of AnchorBoxProcessing with custom decode functions.
// Box decoding
void customBoxDecoding(std::vector&loc, const float* prior, const uint32_t& layer) {
//code
}
//Landmark decoding
void customerLandmarkDecoding(std::vector>& landmark, const float* prior) {
//code
}
AnchorBoxProcessing anchorBoxProc(params, customBoxDecoding, customLandmarkDecoding);Copy to clipboard
**Parent Topic:** [Usage](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-usage.html)
## 12.1.2.3 smartnms Python package
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
The requirements of libAICsmartnms.so (covered in Section [libAICsmartnms.so](https://docs.qualcomm.com/doc/80-PT790-993B/topic/libAICsmartnms-so.html)) must be satisfied to use the smartnms Python package. After that, the Python package must be installed using pip.
Example:
pip install /opt/qti-aic/tools/smart-nms/smartnms-*.whl –force-reinstallCopy to clipboard
The command above installs smartnms, numpy, and matplotlib in the user’s environment. The Python package requires Python 3.8.
from smartnms import SmartNMS
# create nms instance with a config
nms = SmartNMS("/opt/qti-aic/smart-nms/src/examples/configs/user-config-yolov3.yml")
# updating config on the fly
nms.score_threshold = 0.4
nms.nms_threshold = 0.5
nms.profiling_per_iter = True
# api to initializing SmartNMS handlers and validate config parameters
nms.init()
# api to display the final config parameters
nms.display()
# for each inference, we have detections : numpy_array , landmarks : numpy_array, scores : numpy_array
for idx in range(num_inferences):
# getting model outputs from any framework
model_outputs = get_model_output(idx)
detection = { "351": model_outputs[0], "371": model_outputs[1], "391": model_outputs[2] }
result = nms.run(detection, [], [])
results.append(result)
# For plotting CPU efficiency and memory footprints
nms.plot()Copy to clipboard
An example smart-nms plot is shown in [Figure : 1](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html#smartnms_python_package__fig_apps_sdk_smartnms_plot).
Figure : smart-nms plot example
**Parent Topic:** [Usage](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-usage.html)
## 12.1.3 Structure of user-config file
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
The user-config file contains four sections:
- Model architecture information
- I/O information
- Thresholds information
**Parent Topic:** [QAic Smart NMS](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-introduction.html)
## 12.1.3.1 Model architecture information
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
Table : Smart-NMS: user-config file (model architecture)
| No. | Parameter | Required /Optional | Type | Remarks |
| --- | --- | --- | --- | --- |
| 1 | modelname | Required | String | modelname is used to choose the decode the subroutine required for smart-nms.
String options ("resnet34ssd", "mv1ssd", "retinanet", "yolov3", "yolov4", "yolov5", "resnet18ssd", "retinaface") |
| 2 | layout | Required | String | Specifies the layout of output files.
Sstring options (“NAWHC”, “NCHW”, “NHWC”, “NCD”, “NDC”) |
| 3 | num-classes | Required | Int | Denotes the number of classes in the object-detection model. |
| 4 | num-landmark | Optional
Default: 0 | Int | Denotes the number of landmark datapoints present in each branch of landmark output given in the landmark-output-list. |
| 5 | bbox-output-list | Required | List | List of names of output nodes that contain box features. |
| 6 | scores-output-list | Required | List | List of names of output nodes that contain score/logits features. |
| 7 | landmark-output-list | Optional
Default: [] | List | Required if num-landmark > 0, otherwise ignored.
List of names of output nodes that contain score/logits features. |
| 8 | do-class-specific-nms | Optional
Default: False | Boolean | Specify True if class-specific NMS is required. |
| 9 | do-softmax | Optional
Default: False | Boolean | Specify True if softmax needs to be performed on outputs from scores-output-list. |
| 10 | background-class-idx | Optional
Default: 0 | Int | Specify index of class which contains background. |
| 11 | map-classes-coco-81-to-91 | Optional
Default: False | Boolean | Specify True if required to convert class IDs from coco 81 class dataset to coco 91 class dataset. |
| 12 | profiling-per-iter | Optional
Default: False | Boolean | This field enables the profiling of time and memory for each time client call anchor box processing handle. Otherwise, the summary of all the runs will be profiled. |
**Parent Topic:** [Structure of user-config file](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-structure-of-user-config-file.html)
## 12.1.3.2 I/O information
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
The following information is required by the qaic-smart-nms tools for processing of the input/outputfiles. The “`qaic-smart-nms`” tools read the AIC hardware output files from `aic-output-dir`.
| No. | Parameter | Parameter | Parameter | Required /Optional | Type | Remarks |
| --- | --- | --- | --- | --- | --- | --- |
| 13 | model-output-dir | model-output-dir | model-output-dir | Required | PATH | Specify the directory to look for object detection model outputs from AIC/CPU runtime. |
| 14 | model-output-extension | model-output-extension | model-output-extension | Optional
Default: “.bin” | String | Looks for these extension files while reading inference output files from model-output-dir. |
| 15 | model-num-outputs | model-num-outputs | model-num-outputs | Optional
Default: 1 | Int | Process that many numbers of boxes, scores, landmarks from model-output-dir. |
| 16 | abp-output-dir | abp-output-dir | abp-output-dir | Required | PATH | Specify the directory to store smart-nms outputs. |
| 17 | model-run-time | model-run-time | model-run-time | Required | String | "AIC" / "CPU" |
| 18 | aic-model-info | aic-model-info | aic-model-info | | | |
| 19 | | model-binary-dir | model-binary-dir | Required
if model-run-time: "AIC" | PATH | Path of binary directory file generated after compilation of object-detection model. |
| 20 | | skip-transform-value | skip-transform-value | Required
If model-run-time: "AIC"
Default : 5 | Int | Specify the value for different transformation from LRT
Skip Transform Supported Types:
1 = On (initial)
2 = Quantize
3 = Transpose
4 = Convert
5 = Off (Transformed) |
| 21 | | model-output-suffix | model-output-suffix | Optional
Default: “-activation-0-inf-” | String | Append this value to output node names while reading output files from model-output-dir if model-run-time: "AIC". |
| 22 | cpu-model-info | cpu-model-info | cpu-model-info | | | |
| 23 | | model-output-suffix | model-output-suffix | Optional
Default: “” | String | Append this value to output node names while reading output files from model-output-dir if model-run-time: "CPU". |
| 24 | | model-outputs | model-outputs | | | |
| 25 | | | names | Required
If model-run-time: "CPU" | List of String | Names of all output nodes in the model |
| 26 | | | shapes | Required
If model-run-time: "CPU" | List of Lists | Respectively, shapes as a list for each model output, the shape for next model output should be provided in the next line. |
| 27 | | | types | Required
If model-run-time: "CPU" | List of String | List of type model output nodes given in names, respectively.
Supported types "float", "float16", "int8Q", "uint8Q","int16Q", "int32Q", "int32I", "int64I", "bool". |
| 28 | | | offsets | Required
If model-run-time: "CPU" and types is quantized the value will be ignored. | List | List of offsets for model output nodes given in names, respectively.
Offsets and scales are required for ["int8Q","uint8Q", "int16Q", "int32Q"] types. |
| 29 | | | scales | Required
If model-run-time: "CPU" and types is quantized else the value will be ignored. | List | List of scales for model output nodes given in names, respectively.
Offsets and scales are required for ["int8Q","uint8Q", "int16Q", "int32Q"] types. |
| 30 | | model-inputs | model-inputs | | | |
| 31 | | | names | Required
Names of all output nodes in the model. | List | Names of all output nodes in the model. |
| 32 | | | shapes | Required
If model-run-time: "CPU" | List of List | Respectively, shapes as a list for each model output, the shape for next model output should be provided in the next line. |
| 33 | | prior-filepath | prior-filepath | | PATH | Prior refers to a predefined collection of boxes with encoded widths and heights chosen to match the widths and heights of objects in a dataset. With respect to prior information all the final predicted boxes are adjusted.
The Yolo family does not require the prior files.
In that case, an empty path can be provided. |
| 34 | input-info | input-info | input-info | | | |
| 35 | | input-layout | input-layout | Optional | String | Input image layout.
Options for layout ("NCD","NDC","NHWC","NCHW","NAWHC"). |
**Parent Topic:** [Structure of user-config file](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-structure-of-user-config-file.html)
## 12.1.3.3 Thresholds information
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
This section of the user-config file contains information about various thresholds required by Smart-NMS.
Table : Smart-NMS: user-config (ihresholds Information)
| No. | Parameter | Required /Optional | Type | Remarks |
| --- | --- | --- | --- | --- |
| 36 | score-threshold | Optional
Default: 0.05 | Float | Filters boxes with specified score-threshold. |
| 37 | nms-threshold | Optional
Default: 0.45 | Float | Filters boxes with specified threshold on IoU across various boxes. |
| 38 | max-detections-image | Optional
Default: 100 | Int | Limits the maximum detection per batch to specified number. |
| 39 | max-boxes-class | Optional
Default: 100 | Int | Limits the maximum filtered out boxes to specified number per class.
This field will be used when do-class-specific is set to True, otherwise ignored. |
A sample config file for resnet34-ssd that is required from a user to capture all the options.
###################################################################
# Architecture Parameters
##################################################################
model-architecture : "resnet34ssd"
# Options for model architecture :
# "resnet34ssd"
# "mv1ssd"
# "retinanet"
# "yolov3"
# "yolov4"
# "yolov5"
# "resnet18ssd"
# "retinaface"
num-classes : 81
num-landmark : 0
layout : "NCD"
# Input information
input-info:
# Model input image's layout information
input-layout : "NCHW"
# Options for layout :
# "NCD"
# "NDC"
# "NHWC"
# "NCHW"
# "NAWHC"
bbox-output-list : ["325"]
score-output-list : ["329"]
landmark-output-list : []
do-class-specific-nms : True
do-softmax : False
background-class-idx : 0
map-classes-coco-81-to-91 : False
# boolean value to tell which NMSABP is invokedmake
##################################################################
# I/O Configuration
##################################################################
model-output-dir : "./r34-aimet-model/onnx_output"
model-output-extension : ".bin"
model-num-outputs : 1
abp-output-dir : "./r34-new-output"
prior-filepath : "./data/resnet34ssd/priors_r34.bin"
# CPU or AIC
model-run-time : "AIC"
# Required if model-run-time is AIC
aic-model-info:
model-binary-dir: "./r34-aimet-model/r34-binaries"
# 1 = "On",
# 2 = "Quantize",
# 3 = "Transpose",
# 4 = "Convert",
# 5 = "Off",
# Others : "Invalid"
skip-transform-value : 1
model-output-suffix : "-activation-0-inf-"
# Required if model-run-time is CPU
cpu-model-info:
model-output-suffix : ""
model-outputs :
# Array of names
names : ["329" , "325" ]
# 2D Array of shapes
shapes :
- [1, 81, 15130]
- [1, 4, 15130]
# Supported types "float", "float16", "int8Q", "uint8Q", "int16Q", "int32Q", "int32I", "int64I", "bool"
types : ["float","float"]
# offsets and scales are required for ["int8Q", "uint8Q", "int16Q", "int32Q"] types.
# array of offsets
offsets : [0,0]
# array of scales
scales : [1,1]
# Model input config
# All are parallel arrays of input config
model-inputs:
# array of names
names : ["input_1"]
# 2D Array of shapes
shapes :
- [1, 3, 1200, 1200]
##################################################################
# Thresholding Parameters
##################################################################
score-threshold : 0.05
# score threshold to be used in classwise and classagnostic abp-nms
nms-threshold : 0.5
# iou threshold to used in nms
max-detections-image : 600
# overall threshold on one data point/ image
max-boxes-class : 100
# maxBoxesPerClass may be limited to be used with class specific abp-nms
Copy to clipboard
**Parent Topic:** [Structure of user-config file](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-structure-of-user-config-file.html)
## 12.1.4 Configuration files provided with the SDK
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
User-config files that are released as part of SDK are listed.
These can be found at: “`/opt/qti-aic/tools/smart-nms/src/examples/configs/`”.
Table : Configuration files provided with the SDK
| No. | Model | Path to user-config file |
| --- | --- | --- |
| 1 | ResNet34-SSD | `./src/examples/configs/user-config-resnet34ssd.yml` |
| 2 | ResNet34-SSD (aimet) | `./src/examples/configs/user-config-resnet34ssd-aimet.yml` |
| 3 | MobileNetV1-SSD | `./src/examples/configs/user-config-mv1ssd.yml` |
| 4 | ResNet18-SSD | `./src/examples/configs/user-config-resnet18ssd.yml` |
| 5 | RetinaFace | `./src/examples/configs/user-config-retinaface.yml` |
| 6 | Yolov5m | `./src/examples/configs/user-config-yolov5m.yml` |
| 7 | Yolov5s | `./src/examples/configs/user-config-yolov5s.yml` |
| 8 | Yolov3 | `./src/examples/configs/user-config-yolov3.yml` |
| 9 | YoloX | `./src/examples/configs/user-config-yoloxm.yml` |
| 10 | MobileNetV2-SSD | `./src/examples/configs/user-config-mv2ssd.yml` |
| 11 | YoloV4 | `./src/examples/configs/user-config-yolov4.yml` |
| 12 | Effcientdet-d0 | `./src/examples/configs/user-config-efficientdet-d0.yml` |
| 13 | MobileNetV2-SSDLite | `./src/examples/configs/user-config-mv2ssdlite.yml` |
**Parent Topic:** [QAic Smart NMS](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-introduction.html)
## 12.1.5 Limitations
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
Smart-NMS has the following limitations, which will be addressed in future releases of the
SDK.
- Support only 2D-object detection models.
**Parent Topic:** [QAic Smart NMS](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic-smart-nms-introduction.html)
## 12.2 QAic QDetect layers
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
The QDetect layer is a proposed layer provided using Python to import and replace the existing detection layers (YoloLayer, SSDLayer, and so on) in the object detection models. This pluggable layer will eliminate the quantization-sensitive operations and will implement a custom operator that is supported in the AIC backend.
QDetect layers can be applied on a pregenerated ONNX model or can be included in the PyTorch source code itself and registered as a custom op.
**Parent Topic:** [Object detection postprocessing](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
## 12.2.1 Model preparation
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
**Parent Topic:** [QAic QDetect layers](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic_qdetect_layers_introduction.html)
## 12.2.1.1 Using QDetect with pre-exported ONNX model
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
The following steps explain how to modify the model to stitch the QDetect (QNms) plug-in, which will add a QNms node in the graph. The same example can be extended for any object detection algorithm. The example is for the YoloV5s model to generate the model with a Custom QNms node:.
1. Clone the official Ultralytics repo.:
git clone https://github.com/ultralytics/yolov5.gitCopy to clipboard
2. Install the necessary Python packages:
pip install -r /opt/qti-aic/examples/apps/qdetect/qnms/requirements.txtCopy to clipboard
3. Export the ONNX model from the repo.:
cd yolov5 && python export.py –weights yolov5s.pt –include onnxCopy to clipboard
4. Follow the steps in the notebook file to generate the QNms model:
/opt/qti-aic/examples/apps/qdetect/qnms/notebooks/Qualcomm_Cloud_AI_100_QDetect_Demo.ipynbCopy to clipboard
5. The notebook file has the required information to install the Apps and Platform SDKs, and contains the configuration to be passed to the qaic-model-preparator tool at this location:
/opt/qti-aic/examples/apps/qdetect/qnms/notebooks/yolov5_ultralytics_model_info_qdetect.yamlCopy to clipboard
6. Sample qaic-model-preparator command:
python -W ignore qaic-model-preparator.py –config yolov5_ultralytics_model_info_qdetect.yamlCopy to clipboard
A sample config file for yolov5 that is required from a user to capture all the options:
###################################################################
# Model Config Parameters
##################################################################
# Official Model Location: https://github.com/ultralytics/yolov5
# Onnx Ultralytics Model Generation steps:
# git clone https://github.com/ultralytics/yolov3.git
# cd yolov5 && pip install -r requirements.txt
# python export.py --weights yolov3.pt --include onnx # Generate yolov3 onnx model
model:
info:
desc: "YoloV5s Models from Ultralytics Repo."
model_type: "yolov5"
model_path: yolov5s.onnx'
input_info: {"images": [1, 3, 640, 640]}
dynamic_info: False
validate: True
workspace: 'workspace/'
pre_post_handle:
post_plugin: "qdetect" # or "qdetect" or #None
pre_plugin: True
nms_params:
max_output_size_per_class: 100
max_total_size: 100
iou_threshold: 0.65
score_threshold: 0.3
clip_boxes: False
pad_per_class: FalseCopy to clipboard
The output from the exported model would have 4 tensors, and boxes would be in the yxyx format:
detection_boxes : [batch_size, max_total_size, 4]
detection_scores : [batch_size, max_total_size]
detection_classes : [batch_size, max_total_size]
valid_detections : [batch_size]Copy to clipboard
**Parent Topic:** [Model preparation](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic_qdetect_layers_model_preparation.html)
## 12.2.1.2 Using QDetect with model source code
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
The following steps explain how to modify the model to stitch the QDetect plug-in that will add a QNMS node in the graph. The same example can be extended for any object detection algorithm. The example is for the YoloV5s model and shows the steps to generate the model:
1. Clone the official Ultralytics repo:
git clone https://github.com/ultralytics/yolov5.gitCopy to clipboard
2. Install the necessary Python packages:
pip install -r /opt/qti-aic/examples/apps/qdetect/qnms/requirements.txtCopy to clipboard
3. Convert the box coordinates from xywh to yxyx format. For details, refer to:
/opt/qti-aic/examples/apps/qdetect/qnms/pdf_files/ QDetect_Guide_User_Source_Pipeline.pdfCopy to clipboard
4. Skip the concatenation of boxes and scores (typical of Yolo models). For details, refer to:
/opt/qti-aic/examples/apps/qdetect/qnms/pdf_files/ QDetect_Guide_User_Source_Pipeline.pdfCopy to clipboard
5. Modify the \_forward\_once method in such a way that it calls a custom non\_max\_suppression\_aic function instead of non\_max\_suppression, where the necessary mapping of the PyTorch custom layer is defined. The non\_max\_suppression\_aic function in turn creates an instance of class CustomQnmsYolo and uses it to obtain the final post-NMS bounding boxes, scores, and class-indexes.
6. The definition of this custom op and the method to register it are available here:
/opt/qti-aic/examples/apps/qdetect/custom-op/qnmsExample/utils/yolo_qnms_plugin.pyCopy to clipboard
7. A sample script to export a new model with a QNms custom operator is available here:
/opt/qti-aic/examples/apps/qdetect/qnms/user_source_file/samples/sample_yolo_export.pyCopy to clipboard
Note: For more details on CustomQnmsYolo custom op, see the Compiler Library Reference Manual.
**Parent Topic:** [Model preparation](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic_qdetect_layers_model_preparation.html)
## 12.2.2 Compilation using qaic-exec
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
Qnms models follow the usual compilation process and commands. It is not required to
provide additional options.
**Parent Topic:** [QAic QDetect layers](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic_qdetect_layers_introduction.html)
## 12.2.3 Limitations
Source: [https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html](https://docs.qualcomm.com/doc/80-PT790-993B/topic/Object-Detection-Post-processing.html)
In the current release, YoloV2, YoloV3, YoloV4, YoloV5, YoloV7, Resnet34-SSD, RetinaNet,
EfficientDet, and MobileNetV1-SSD are supported.
**Parent Topic:** [QAic QDetect layers](https://docs.qualcomm.com/doc/80-PT790-993B/topic/qaic_qdetect_layers_introduction.html)
Last Published: Jul 26, 2023
[Previous Topic
Appendix – Model Architecture](https://docs.qualcomm.com/bundle/publicresource/80-PT790-993B/topics/end-to-end-workflow-qinference-optimizer.md#end-to-end-workflow-qinference-optimizer_qinference-optimizer-appendix-model-architecture)