# Parallel AI fusion
Source: [https://docs.qualcomm.com/doc/80-70017-50/topic/gst-ai-parallel-inference.html](https://docs.qualcomm.com/doc/80-70017-50/topic/gst-ai-parallel-inference.html)
The **gst-ai-parallel-inference** application enables you to perform object
detection, object classification, pose detection, and image segmentation on an input stream
from different sources such as a camera, a file, or an RTSP network. The use cases use the
LiteRT models for object detection, image segmentation, classification and pose
detection.
Note: This application is not supported on QCS8275.
The figure shows the pipeline, which takes the input from a camera, file, or an RTSP
stream, performs the parallel inferencing for the four use cases, and display the
results side by side on the screen.
For information on the plugins used in the pipeline flow, see [Pipeline flow](https://docs.qualcomm.com/doc/80-70017-50/topic/gst-ai-parallel-inference.html#gst-ai-parallel-inference__section_gcg_r3s_lbc).
Figure : gst-ai-parallel-inference pipeline
Learn how to enhance your development projects by performing the four parallel AI
inferences on a live camera stream. This video shows the step-by-step procedure on how
to set up the pipeline and display the results side by side. The will open in a new
tab.
## Prerequisites
- [Download model, label and config JSON files for LiteRT and Qualcomm AI Engine direct](https://docs.qualcomm.com/doc/80-70017-50/topic/ai-ml-sample-applications.html#ai-ml-sample-applications__section_fsl_lgz_scc).
- To access your host device, enable SSH. For instructions, see [Log in using SSH](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-254/how_to.html#use-ssh).
Note: If SSH is already enabled, you can skip this
step.
- Enter the SSH shell and run the use cases:
ssh root@Copy to clipboard
- Enable the
display:
export XDG_RUNTIME_DIR=/dev/socket/weston && export WAYLAND_DISPLAY=wayland-1Copy to clipboard
- Push the files from the Linux
host:
scp root@:/opt/Copy to clipboard
- Push the video.mp4 file in opt
folder.
## Run the application
For sample model and label files, see [Sample model and label files](https://docs.qualcomm.com/doc/80-70017-50/topic/gst-ai-parallel-inference.html#gst-ai-parallel-inference__section_pnn_hmb_4dc).
Enter SSH shell and copy the YOLO-NAS label files to YOLOv8:
cp /opt/yolonas.labels /opt/yolov8.labelsCopy to clipboard
To run the application, use the following syntax:
gst-ai-parallel-inference [OPTION?]Copy to clipboard
- Run with input from the primary and secondary
cameras:
gst-ai-parallel-inference -c 0Copy to clipboard
gst-ai-parallel-inference -c 1Copy to clipboard
- Run with input from a
file:
gst-ai-parallel-inference -s /opt/video.mp4Copy to clipboard
- Run with input from an RTSP
stream:
gst-ai-parallel-inference --rtsp-ip-port="rtsp:///live.mkv"Copy to clipboard
- Run with input from a video stream with constants for the LiteRT models for all
four
inferences:
gst-ai-parallel-inference --file-path="/opt/video.mp4" --object-detection-constants="YOLOv8,q-offsets=<21.0, 0.0, 0.0>,q-scales=<3.093529462814331,0.00390625, 1.0>;" --pose-detection-constants "Posenet,q-offsets=<8.0>,q-scales=<0.0040499246679246426>;" --segmentation-constants="deeplab,q-offsets=<0.0>,q-scales=<1.0>;" --classification-constants="Mobilenet,q-offsets=<-95.0>,q-scales=<0.18740029633045197>;"Copy to clipboard
To display the available help options, run the following command in the SSH
shell:
gst-ai-parallel-inference -hCopy to clipboard
To stop the use case, press CTRL + C.
## Expected output
After performing the four parallel inferences, the results are displayed side by side
on the screen.
Figure : Expected output for gst-ai-parallel-inference application
## Pipeline flow
The table lists the plugins used in the parallel inference pipeline:| Plugin | Description |
| --- | --- |
| Camera source:[qtiqmmfsrc](https://docs.qualcomm.com/doc/80-70017-50/topic/qtiqmmfsrc.html) |
Captures the live stream from camera.
Uses tee to split the stream for inferencing.
|
| File source: filesrc |
Captures the video stream using filesrc, followed by qtdemux, which demultiplexes the stream.
Uses tee to split the stream for inferencing.
|
| RTSP source: rtspsrc |
Captures the RTSP stream using rtspsrc, followed by rtph264depay for video extraction.
Uses tee to split the stream for inferencing.
|
| h264parse | Parses the H.264 video. |
| [v4l2h264dec](https://docs.qualcomm.com/doc/80-70017-50/topic/v4l2h264dec.html) | Decodes the video |
| [qtimlvconverter](https://docs.qualcomm.com/doc/80-70017-50/topic/qtimlvconverter.html) |
Receives the video stream on its sink pad.
Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input.
Color conversion
Scaling (up or down)
Normalization
The tensor stream is used for inferencing in the later stages of the pipeline. |
| [qtimltflite](https://docs.qualcomm.com/doc/80-70017-50/topic/qtimltflite.html) |
After the inference runtime receives the tensor stream on its sink pad, it runs the inference.
Produces a tensor stream with the inference results on its source pad.
|
| Postprocessing plugins | Handles the inference results from any object detection, classification, pose detection, and segmentation model.
Applies the threshold to the chosen number of results.
Loads the corresponding modules for various pose estimation models. For the use cases described in this section, qtimlvpose loads the PoseNet module.
Produces results as video frames with poses drawn, sending them to the sink pad of the qtivcomposer.
qtimlvsegmentation: Converts the inference tensors that it receives on its sink pad into video formats that the multimedia plugins for further processing.