# Multi model daisychain detection classification The use case uses a sequential pipeline design to detect objects and classify up to four of the detected objects. Note This section isn’t applicable for the QLI 2.0 RC2 release. The following figure shows the flow of the use case execution: **Figure : Pipeline for multi model daisychain detection classification** This pipeline involves two sequential stages, each leveraging a set of specialized plugins to process video frames and extract meaningful information. 1. Object detection: 1. The filesrc plugin facilitates reading video data from a file. The decoder plugins then convert the video into raw formats such as NV12 (YUV). 2. tee splits the NV12 video into two streams. qtimlvconverter and qtimetamux receive one video stream each. 3. The qtimlvconverter sequentially processes the NV12 video frame and converts it into RGB tensor of 640 × 640. 4. The qtimltflite performs inference using the YOLOX model, generating three output tensors that are further split using tee. The YOLOv8 module processes one stream to produce bounding boxes. Another stream attaches to the qtimetamux, which further performs classification. 2. Object classification: 1. The NV12 video from the qtimetamux is further split using tee. One stream attaches to the original video frame using the qtivcomposer. The other frame uses the qtivsplit to split the stream into total four output streams, indicating a maximum of four detected objects classified in a frame. 2. The qtimlvconverter preprocesses all the four streams to the convert the NV12 frame into a RGB tensor of dimension 224 × 224. The qtimltflite performs inference using the Inceptionv3 model and generates single output tensor for all the four video streams in parallel. 3. The qtimlpostprocess parses the output tensor of all four video streams using MobileNet-softmax module to produce classification labels, which attach to the original video using the qtivcomposer. 4. The Waylandsink renders video frames with ML overlays directly on the screen using Wayland display server. Run the use case on the target device: export XDG_RUNTIME_DIR=/run/user/1000 && export WAYLAND_DISPLAY=wayland-1 Copy to clipboard gst-launch-1.0 -e \ qtimltflite name=tflite_detect delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=/etc/models/yolox_quantized.tflite \ qtimltflite name=tflite_Mobilenet_1 delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=/etc/models/inception_v3_quantized.tflite \ qtimltflite name=tflite_Mobilenet_2 delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=/etc/models/inception_v3_quantized.tflite \ qtimltflite name=tflite_Mobilenet_3 delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=/etc/models/inception_v3_quantized.tflite \ qtimltflite name=tflite_Mobilenet_4 delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=/etc/models/inception_v3_quantized.tflite \ qtimlvconverter name=ml_convert_0 ! queue ! tflite_detect. tflite_detect. ! queue ! tee name=t_split_1 \ qtivcomposer name=mixer \ sink_0::position="<0, 0>" sink_0::dimensions="<1280, 720>" \ sink_1::position="<0, 0>" sink_1::dimensions="<1280, 720>" \ sink_2::position="<0, 0>" sink_2::dimensions="<384, 216>" \ sink_3::position="<896, 0>" sink_3::dimensions="<384, 216>" \ sink_4::position="<0, 504>" sink_4::dimensions="<384, 216>" \ sink_5::position="<896, 504>" sink_5::dimensions="<384, 216>" \ sink_6::position="<0, 0>" sink_6::dimensions="<384, 40>" \ sink_7::position="<896, 0>" sink_7::dimensions="<384, 40>" \ sink_8::position="<0, 504>" sink_8::dimensions="<384, 40>" \ sink_9::position="<896, 504>" sink_9::dimensions="<384, 40>" \ mixer. ! waylandsink fullscreen=true \ filesrc location=/etc/media/video.mp4 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=dmabuf output-io-mode=dmabuf ! video/x-raw,format=NV12 ! queue ! tee name=v_split_1 ! queue ! metamux1. v_split_1. ! queue ! ml_convert_0. \ t_split_1. ! queue ! qtimlpostprocess settings="{\"confidence\": 75.0}" results=4 module=yolov8 labels=/etc/labels/yolox.json ! text/x-raw ! queue ! qtimetamux name=metamux1 ! queue ! tee name=t_split_2 ! queue ! mixer. \ t_split_1. ! queue ! qtimlpostprocess settings="{\"confidence\": 75.0}" results=4 module=yolov8 labels=/etc/labels/yolox.json ! video/x-raw,width=512,height=288 ! queue ! mixer. \ t_split_2. ! queue ! qtivsplit name=vsplit1 src_0::mode=single-roi-meta src_1::mode=single-roi-meta src_2::mode=single-roi-meta src_3::mode=single-roi-meta \ vsplit1. ! queue ! tee name=split_1 ! queue ! ml_convert_1. split_1. ! queue ! mixer. \ vsplit1. ! queue ! tee name=split_2 ! queue ! ml_convert_2. split_2. ! queue ! mixer. \ vsplit1. ! queue ! tee name=split_3 ! queue ! ml_convert_3. split_3. ! queue ! mixer. \ vsplit1. ! queue ! tee name=split_4 ! queue ! ml_convert_4. split_4. ! queue ! mixer. \ qtimlvconverter name=ml_convert_1 ! queue ! tflite_Mobilenet_1. tflite_Mobilenet_1. ! queue ! mlclass_1. \ qtimlvconverter name=ml_convert_2 ! queue ! tflite_Mobilenet_2. tflite_Mobilenet_2. ! queue ! mlclass_2. \ qtimlvconverter name=ml_convert_3 ! queue ! tflite_Mobilenet_3. tflite_Mobilenet_3. ! queue ! mlclass_3. \ qtimlvconverter name=ml_convert_4 ! queue ! tflite_Mobilenet_4. tflite_Mobilenet_4. ! queue ! mlclass_4. \ qtimlpostprocess name=mlclass_1 settings="{\"confidence\": 60.0}" results=3 module=mobilenet-softmax labels=/etc/labels/classification.json ! video/x-raw,width=384,height=40 ! queue ! mixer. \ qtimlpostprocess name=mlclass_2 settings="{\"confidence\": 60.0}" results=3 module=mobilenet-softmax labels=/etc/labels/classification.json ! video/x-raw,width=384,height=40 ! queue ! mixer. \ qtimlpostprocess name=mlclass_3 settings="{\"confidence\": 60.0}" results=3 module=mobilenet-softmax labels=/etc/labels/classification.json ! video/x-raw,width=384,height=40 ! queue ! mixer. \ qtimlpostprocess name=mlclass_4 settings="{\"confidence\": 60.0}" results=3 module=mobilenet-softmax labels=/etc/labels/classification.json ! video/x-raw,width=384,height=40 ! queue ! mixer. Copy to clipboard To stop the use case, use **CTRL + C**. The following table provides the sequential processing stages of the pipeline execution: | Process | Description | | --- | --- | | filesrc |
Reads data from a file and pushes it downstream in the pipeline.
Uses location property to set the path of file to read.
Parses and prepares the H.264 stream for decoding.
Adds necessary headers and ensures proper frame alignment.
Uses V4L2 interface and acts as a hardware accelerated H.264 decoder.
Uses DMA buffer to share the memory.
transforms the incoming video buffer into neural network tensors.
Performs necessary format conversion and resizing.
Loads and runs the LiteRT model.
Uses delegate property to set on which core model inference is executed.
Has model property to set path of model.
Post-processes the tensor output of the inference plugin.
Parses the tensor and generates the list of predicted output nodes.
Uses GPU hardware to merge several input video streams into single output stream.
Uses position and dimension properties to set the co-ordinates and size of the destination rectangle.
Renders video frames to a display using the Wayland protocol.
Acts as an ouptut component in the gstreamer pipeline for systems running on Wayland.