# Qualcomm Neural Processing SDK for AI
Source: [https://docs.qualcomm.com/doc/80-88500-4/topic/144_Qualcomm_Neural_Processing_SDK_for_AI.html](https://docs.qualcomm.com/doc/80-88500-4/topic/144_Qualcomm_Neural_Processing_SDK_for_AI.html)
The Qualcomm Neural Processing SDK for AI is a software accelerated inference-only runtime engine for the execution of deep neural networks.
It provides abstraction to underlying details on the model execution, and provides flexibility to design and develop machine learning application pipelines with ease.
With the SDK, you can do the following:
- Execute an arbitrarily deep neural network
- Execute the network on the Snapdragon CPU, the Adreno GPU, or the Hexagon DSP, and HTA
- Debug the network execution on the x86 processor architecture
- Convert the Caffe and Caffe2, ONNX, and TensorFlow, TensorFlow Lite, and PyTorch models to a Qualcomm Neural Processing SDK deep learning container (DLC) file
- Quantize the DLC files to 8-bit fixed point for running on the AI accelerators (Qualcomm Hexagon Vector
eXtensions and Hexagon Tensor Accelerator)
- Debug and analyze the performance of the network with Qualcomm Neural Processing SDK tools
- Integrate a network into applications and other code via C++ or Java
A generic workflow involving the Qualcomm Neural Processing SDK runtime engine is shown the following example:
1. The developers develop a model, train the model to meet the requirements as shown in following figure (training phase).
2. After the model is trained and frozen, the model with static weights and biases is converted to the native format (DLC) of the Qualcomm Neural Processing SDK.
3. After a model file DLC is generated, one can use the optional offline optimization tools to optimize the model. These optimizations may include quantization and compression techniques.
4. A developer writes a machine learning application (Qualcomm Neural Processing SDK-enabled application) using the Qualcomm Neural Processing SDK C++/Java API or GStreamer Qualcomm Neural Processing SDK plugin (qtimlsnpe) to quickly execute the converted model.
Note: This process may involve additional steps to debug the model performance and accuracy. The debugging tools are provided as part of the Qualcomm Neural Processing SDK.
Figure : Model to runtime workflow using Qualcomm Neural Processing SDK
Note: For more information on the Qualcomm Neural Processing SDK and its developmental workflow, see Snapdragon Neural Processing Engine SDK Reference Guide (80-NL315-14) available on the Qualcomm Developer Network.
The developers can quickly start integrating their machine learning model by using the already available GStreamer plugins by supplying the converted DLC model.
The GST plugin does the following functions directly:
- Loads the .dlc file (Qualcomm Neural Processing SDK model)
- Preprocessing and postprocessing of video frames
- Ability to configure the Qualcomm Neural Processing SDK to run on the DSP or CPU or GPU or HTA
- Abstraction from the Qualcomm Neural Processing SDK
Figure : Qualcomm Intelligent Multimedia SDK workflow
- Upon the GST launch, the inference frames from either a camera source (YUV) or a file source are delivered to the qtimlsnpe sink along with a model DLC (generated following the Qualcomm Neural Processing SDK workflow).
- The qtimlsnpe sink will in turn use the Qualcomm Neural Processing SDK runtime to offload the model computation to the requested runtime (DSP, GPU, or CPU).
- Inference results are gathered back in qtimlsnpe sink for postprocessing, for example, overlaying bounding boxes and class IDs on a detected object in frame, if the model is an object detection model.
The following example using GST shows the use of inferencing via a live camera 1080p stream with a bounding box overlay applied inline on the YUV Stream. The render is on Weston display; hence, XDG\_RUNTIME\_DIR should be set.
Push the corresponding labels and models in the folders you refer to in the labels and config variables respectively:
The following use case uses the You Only Look Once (YOLOv5) computer vision model. YOLOv5 is used for detecting objects. The command uses a YOLOv5 model called yolov5s\_relu\_finetune\_quantized\_cle\_bc, which is used to identify an object in scene from a video stream.
export XDG_RUNTIME_DIR=/run/user/root && gst-launch-1.0 -e --gst-debug=2 \
qtiqmmfsrc name=camsrc ! video/x-raw\(memory:GBM\),format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! tee name=split \
split. ! queue ! qtimetamux name=metamux ! queue ! qtioverlay text-font-size=24 ! queue ! waylandsink sync=false fullscreen=true \
split. ! queue ! qtimlvconverter ! queue ! qtimlsnpe delegate=dsp model=/data/yolov5s_relu_finetune_quantized_cle_bc.dlc layers="" ! queue ! qtimlvdetection threshold=90.0 results=10 module=yolov5s labels=/data/yolov5s.labels ! text/x-raw ! queue ! metamuxCopy to clipboard
1. The yolov5s\_relu\_finetune\_quantized\_cle\_bc model detects an object from the video stream.
2. The stream is coming through a camera source and overlaying bounding boxes over the detected objects using overlaylib. The results are displayed on a local display device.
3. The video stream is collected from the camera source plugin and two copies are created:
- One is sent to the qtimetamux plugin to retain the video stream
- Another is sent to the ML inferencing pipeline
4. The qtimlvconverter preprocessing plugin receives the video stream on its sink pad. It does the following:
1. Implements the preprocessing operations such color conversion, scaling down/up, and normalization on the stream data when model expects floating point values as input.
2. Converts the video stream to a tensor stream on the source pad, which the yolov5s\_relu\_finetune\_quantized\_cle\_bc model can use for inferencing in a later part of the pipeline.
5. The qtimlsnpe ML inferencing plugin does the following:
1. Loads the model and modifies the graph for the chosen delegate.
2. Receives the tensor stream on its sink pad and executes the inference.
3. Produces the tensor stream with the object detection results on its source pad.
6. The qtimlvdetection postprocessing plugin works on the inference results from any object detection model. It does the following:
- Converts the inference tensors it receives on its sink pad into video or text format, which the multimedia plugins can understand.
- Applies the threshold to the chosen number of results.
- Loads the corresponding modules for a variety of detection models.
Here in this use case, the qtimlvdetection loads the YOLOv5 sub module and produces results as structures of text and sends them to the sink pad of metamux plugin.
7. The metamux plugin receives the video stream and text stream of the bounding box results corresponding to the video stream on its sink pads.
1. It produces GST buffers with contents of the video stream from its sink pad.
2. It adds bounding boxes as GstVideoRegionOfInterest from data sink pad to the GST buffer meta (meta muxing) on its source pad.
8. The qtioverlay plugin receives the muxed stream and overlays the bounding boxes on the video frame using computing language (CL) and produces GST buffers with overlays in its source pad.
9. The waylandsink submits the video stream it is receiving on its sink pad to Weston. Weston renders the video stream on a local display device.
10. The video stream captured by the camera source plugin and the bounding boxes generated for the objects in that scene are displayed on the local display device.
**Parent Topic:** [Artificial intelligence/Machine learning](https://docs.qualcomm.com/doc/80-88500-4/topic/143_AI_ML.html)
Last Published: Aug 18, 2023
[Previous Topic
Artificial intelligence/Machine learning](https://docs.qualcomm.com/bundle/publicresource/80-88500-4/topics/143_AI_ML.md) [Next Topic
TensorFlow](https://docs.qualcomm.com/bundle/publicresource/80-88500-4/topics/145_TensorFlow.md)