# Multistream inference
Source: [https://docs.qualcomm.com/doc/80-70022-50/topic/multistream-inference.html](https://docs.qualcomm.com/doc/80-70022-50/topic/multistream-inference.html)
The **gst-ai-multistream-inference** application shows AI inference (object
detection and classification) on up to 32 input streams coming from camera, file, or RTSP
stream.
The following figure shows the pipeline, which receives several input streams,
preprocesses them, runs AI inferences, combines the streams, and merges them all into a
single video output.
The maximum number of input streams supported on each SoC as verified on 1080P and 720P
are follows:
- QCS6490–8
- Dragonwing IQ-8275–16
- Dragonwing IQ-9075–32
Note: Dragonwing IQ-8275 isn't supported on the Ubuntu
Server.
The output is displayed on an HDMI display, saved as an H.264 encoded MP4 file, or
converted into an RTSP stream.
For information about the plugins used in this pipeline, see [Pipeline flow](https://docs.qualcomm.com/doc/80-70022-50/topic/multistream-inference.html#multistream-inference__section_opk_tn2_ncc).
Figure : gst-ai-multistream-inference pipeline
Qualcomm
Open source
File (default) Stream 1 Camera (optional) RTSP (optional)
qtivcomposer
tee
tee
qtdemux h264parse V4l2h264dec qtimlvconverter qtimltflite/ qtimlpostprocess
filesrc
or or
waylandsink
filesink
qtirtspbin
qtivcomposer
qtimlvconverter qtimltflite qtimlpostprocess
qtiqmmfsrc
or or
waylandsink
filesink
qtirtspbin
qtivcomposer
rtph264 depay h264parse V4l2h264dec tee qtimlvconverter qtimltflite qtimlpostprocess rtspsrc
or or
waylandsink
filesink
qtirtspbin
File (default) Stream 32 Camera (optional) RTSP (optional)
qtivcomposer
tee
tee
qtdemux h264parse V4l2h264dec qtimlvconverter qtimltflite/ qtimlpostprocess
filesrc
or or
waylandsink
filesink
qtirtspbin
qtivcomposer
qtimlvconverter qtimltflite qtimlpostprocess
qtiqmmfsrc
or or
waylandsink
filesink
qtirtspbin
qtivcomposer
rtph264 depay h264parse V4l2h264dec tee qtimlvconverter qtimltflite qtimlpostprocess rtspsrc
or or
waylandsink
filesink
qtirtspbin
Stream 2
Stream 31
## Sample model and label files
Table : Sample model and label files for gst-ai-multistream-inference
| Runtime | Model files | Label files |
| :--- | :--- | :--- |
| LiteRT | inception_v3_quantized.tflite yolox_quantized.tflite yolox.json classification.json ",
""
],
"input-type": "",
"model": "",
"labels": "",
"output-display": "<0 or 1>",
"output-file-path": "",
"output-ip-address": "",
"output-port-number": "",
"use-case": "<0 or 1>"
}Copy to clipboard
For example, run the application for object detection on 8
H.264 file inputs and view the output on a Wayland
display:
{
"input-file-path":
[
"/etc/media/video.mp4",
"/etc/media/video.mp4",
"/etc/media/video.mp4",
"/etc/media/video.mp4",
"/etc/media/video.mp4",
"/etc/media/video.mp4",
"/etc/media/video.mp4",
"/etc/media/video.mp4"
],
"model": "/etc/models/yolox_quantized.tflite",
"labels": "/etc/labels/yolox.json",
"input-type": "h264",
"output-display": 1,
"use-case": 0
}Copy to clipboard
4. Run the gst-ai-multistream-inference application:
gst-ai-multistream-inference --config-file=/etc/configs/config-multistream-inference.jsonCopy to clipboard
Note: If a drop in performance is observed, you can use
YOLOv8 LiteRT model. For YOLOv8 export instructions, see Step 6 in [Prerequisites](https://docs.qualcomm.com/doc/80-70022-50/topic/download-model-and-label-files.html).
5. To display the available help options, run the following commands in the SSH
shell:
gst-ai-multistream-inference -hCopy to clipboard
6. To stop the use case, use CTRL + C.
## Expected output
Figure : Expected output for gst-ai-multistream-inference
## Pipeline flow
The following table lists the plugins used in the multistream inference
pipeline:| Plugin | Description |
| --- | --- |
| Camera source:[qtiqmmfsrc](https://docs.qualcomm.com/doc/80-70022-50/topic/qtiqmmfsrc.html) | Captures the live stream from camera. Uses tee to split the stream for inferencing. Captures the video stream using filesrc, followed by Uses tee to split the stream for inferencing. Captures the RTSP stream using rtspsrc, followed by Uses tee to split the stream for inferencing. Receives the video stream on its sink pad. Performs the following preprocessing on the stream data.Color conversion Scaling (up or down) Normalization After the inference runtime receives the tensor stream on Produces a tensor stream with the inference results on its qtimlpostprocess for detection use case:Applies a threshold to the chosen number of Loads the YOLOv8 module. Produces video frames with only bounding boxes that qtimlpostprocess for classification use caseApplies the threshold to the chosen number of Loads the MobileNet-softmax module. Produces results as video frames with classification Composes frames with contents from its sink pads. Pushes the GStreamer buffers containing these composed Waylandsink submits the video stream received on its sink Weston renders the video stream on a local display. Receives the video stream on its sink pad. Saves the stream as a H.264-encoded MP4 file. Serves as a network sink. Transmits UDP packets to the network. num-camera: The number of inputs from thecamera: The input camera ifnum-camera=1.input-file-path: The directory path to theinput-rtsp-path: The address of the RTSPrtsp://<ip>:<port>/<stream> output-file-path: The directory path tooutput-ip-address: The IP address of theoutput-port-number: The port number of theoutput-display: The connected display