# GStreamer plugin architecture The Qualcomm IM SDK hides the complexity of the hardware within the plugin architecture and provides APIs to applications. Using this framework, you can create applications without the need to access the low-level platform libraries and hardware details, which can vary across platforms. The [Configure Qualcomm GStreamer plugins](https://docs.qualcomm.com/doc/80-80021-50/topic/qim-sdk-plugins.html) include video encoding/decoding, camera ISP, GPU, display, audio DSP (aDSP), and AI/ML accelerators. **Figure : Qualcomm IM SDK GStreamer plugin architecture** ## Graphics and display architecture The Wayland server uses the Wayland protocol to enable communication between the display and graphics servers, and its clients. The display and graphics architecture uses the Weston server (Weston). Weston manages both composition and display while it runs as a separate process in the system. **Figure : Weston/Wayland architecture** Weston/Wayland components | Component | Description | | --- | --- | | Wayland/ GLES client | Implements the Wayland protocol for Weston. | | Weston server | Implements the Wayland compositor. - Uses the Kernel Mode Setting (KMS) to set up the display. - Performs the compositing using OpenGL ES and Direct Rendering Manager (DRM). | | Secure dynamic messaging (SDM) back-end |

Uses display hardware abstraction layer (HAL) to interact with the display hardware.

Provides several platform-dependent implementations with one of it designated for DRM and KMS.

| | GBM buffer management library (libGBM) |

Includes DMA back-end.

Used for zero-copy buffer sharing between display and graphics.

| | EGL platform-specific driver (EGL sub-driver) | Interacts with GBM and Wayland protocol to communicate with the Weston compositor. | ## Camera architecture The camera architecture provides a high-level abstraction on how the underlying camera driver and hardware interact with the [qtiqmmfsrc](https://docs.qualcomm.com/doc/80-80021-50/topic/qtiqmmfsrc.html) camera plugin. The following figure shows the camera architecture, where the qtimmfsrc plugin acts as a client to the Qualcomm Camera Service. **Figure : Camera pipeline** Camera pipeline components | Component | Description | | --- | --- | | Linux embedded (LE) camera service |

Runs as a daemon in the system:

Provides easy remote procedure call (RPC) APIs to control the camera.

Exposes helper client APIs, which perform RPC between client and server.

Uses the HAL3 API, which interacts with the camera back

end (CamX) and camera driver to configure camera sensor and image signal processor (ISP) hardware.

Uses GBM to assign buffers for each camera stream and submits them to HAL3.

| | Client/server architecture | Allows the multi-client and multi-camera use cases to do the following:

Create many instances of qtiqmmfsrc where one instance of the plugin corresponds to one physical or logical camera.

Implement camera use cases. These instances can be in the same process or different processes depending upon the use case

| | qtiqmmfsrc |

Provides several streams in parallel (many source pads), and each stream can be of different formats (NV12/21 or MJPEG).

For example, if there are three parallel streams, they’re processed as follows:

One stream for local storage.

One stream for YUV stream for live camera preview on a local display.

One for network streaming.

Supports ‘N’ number of parallel streams, where ‘N’ depends on camera ISP capability.

| | [Waylandsink](https://docs.qualcomm.com/doc/80-80021-50/topic/waylandsink.html) |

Uses the raw YUV (NV12/21) stream output to render the camera frames to a physical display.

Achieves a live camera preview use case.

Allows the postprocessing element to improve the quality.

Allows the ML inferencing elements to do inferencing on a live camera.

| | Buffers |

The same buffers are circulated to the clients (qtiqmmfsrc) and other plugins in the pipeline.

The buffer pools manage the buffers to avoid per-frame allocation.

| ### Related information [Camera use cases](https://docs.qualcomm.com/doc/80-80021-50/topic/camera.html) ## Video architecture The [v4l2h264enc](https://docs.qualcomm.com/doc/80-80021-50/topic/v4l2h264enc.html) and [v4l2h265enc](https://docs.qualcomm.com/doc/80-80021-50/topic/v4l2h265enc.html) video encode plugins connect with the camera plugin to implement the advance video encode (H.264 or H.265) use cases. The [v4l2h264dec](https://docs.qualcomm.com/doc/80-80021-50/topic/v4l2h264dec.html) and [v4l2h265dec](https://docs.qualcomm.com/doc/80-80021-50/topic/v4l2h265dec.html) video decode plugins connect with Waylandsink to implement the video playback use cases. ### Encode The following figure shows the architecture for video encoding. The v4l2h264enc and v4l2h265enc plugins are connected to a file multiplexer (MP4 or MPEGTS) to save the encode bitstream to the file system. For video encode use cases, see [Video encode and decode](https://docs.qualcomm.com/doc/80-80021-50/topic/camera-and-video-encode.html). **Figure : Video encode pipeline** Video encode components | Components | Description | | --- | --- | | qtiqmmfsrc | Captures and provides several video streams in parallel. For more information about the camera service, see Camera architecture. | | Video encode plugins | Uses the V4L2 driver and APIs to encode the video stream. - [v4l2h264enc](https://docs.qualcomm.com/doc/80-80021-50/topic/v4l2h264enc.html): Encodes the video for AVC (H.264) format. - [v4l2h265enc](https://docs.qualcomm.com/doc/80-80021-50/topic/v4l2h265enc.html): Encodes the video for HEVC (H.265) format. | | H.264/H.265 parse | Parses the video stream. | | Filemux | Multiplexes encoded video stream and saves to a file. | ### Decode The following figure shows the architecture for video decoding where the video stream from a file is demultiplexed, decoded, and displayed on a local device. The decode parameters are exposed as a property to the application to provide control over the video decode pipeline. **Figure : Video decode pipeline** For video decode use cases, see [Video playback use cases](https://docs.qualcomm.com/doc/80-80021-50/topic/video-playback-use-cases.html). Video decode components | Components | Description | | --- | --- | | Video decode plugins | Uses the V4L2 driver and APIs to decode the video stream and connects with the Waylandsink plugin. - [v4l2h264dec](https://docs.qualcomm.com/doc/80-80021-50/topic/v4l2h264dec.html): Decodes the video for AVC (H.264) format. - [v4l2h265dec](https://docs.qualcomm.com/doc/80-80021-50/topic/v4l2h265dec.html): Decodes the video for HEVC (H.265) format. | | Waylandsink |

Receives the GBM buffers (decoded buffers) as an input.

Sends them to the Weston server (through Wayland protocol) for display composition.

| ## Audio architecture For audio capture and playback, use the [pulsesrc](https://docs.qualcomm.com/doc/80-80021-50/topic/pulsesrc.html) and [pulsesink](https://docs.qualcomm.com/doc/80-80021-50/topic/pulsesink.html) GStreamer plugins. You can encode and decode audio through an open-source software. ### Capture The following figure shows the audio capture using the pulsesrc plugin. The audio stream is then processed and written to a file. **Figure : Audio capture pipeline** Audio capture components | Component | Description | | --- | --- | | Pulsesrc |

Captures the audio and shares it with the underlying PulseAudio server.

Sets the audio source from which the pulse-code modulation (PCM) audio samples are captured.

Saves the PCM samples to a file.

| | PulseAudio server |

Interacts with the Qualcomm hardware.

Uses a pluggable module to interact with the ALSA driver to process the audio data.

| ### Playback Use the pulsesink plugin for audio playback. It’s an upstream plugin that allows you to play audio from various audio sources (live source or an encoded audio file). **Figure : Audio playback pipeline** Audio playback components | Component | Description | | --- | --- | | Pulsesink |

Interacts with the underlying PulseAudio server.

Sets the audio playback sink on which the audio is played.

Plays only the PCM data.

| | PulseAudio server |

Interacts with the Qualcomm hardware.

Uses a pluggable module to interact with the ALSA driver to process the audio data for playback.

| ### Encode You can encode the audio using an open-source audio encoder plugin. **Figure : Audio encoding pipeline** Audio encoding components | Component | Description | | --- | --- | | Pulsesrc | Captures the audio and shares it with the underlying PulseAudio server. | | PulseAudio server | Interacts with the ALSA driver to process the audio data for encoding. | | Encode | Encodes the audio data using an open-source software. | | Parse | Parses the audio data. | | Filemux | Multiplexes it and saves it in an MP4 or MPEGTS container. | ### Decode You can decode the audio using an open-source audio decoder plugin. **Figure : Audio decode pipeline** Audio decode components | Component | Description | | --- | --- | | Filesrc | Reads the audio data. | | Stream demux | Demultiplexes the audio data. | | Decode | Decodes the audio data using an open-source software. | | Pulsesink | Interacts with the underlying PulseAudio server and plays the decoded audio. | | PulseAudio server | Interacts with the ALSA driver to process the audio data for decoding. | ## Machine learning architecture Qualcomm IM SDK supports machine learning use cases that include video preprocessing, model inference, output tensor postprocessing, and inference result overlay to stream. The machine learning framework provides the following types of video analytics: - Image classification - Object detection - Image segmentation With machine learning plugins in the framework, use the following for inferencing: - [LiteRT](https://ai.google.dev/edge/litert) (or TFLite) - [Qualcomm Neural Processing SDK](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-2/overview.html) For each of the engines, the delegates speed up the model inference performance. ### ML pipeline processing The following figure depicts the pipeline in which the captured video/audio stream is processed among the preprocessing, inference, and postprocessing plugins. **Figure : ML pipeline** The following table lists the sequential stages during which the video/audio stream is captured, processed, and the output is either displayed on a screen or saved to a file. ML pipeline processing stages | Process | Description | | --- | --- | | Source |

qtiqmmfsrc captures the YUV streams from live camera source.

The filesrc plugin captures any format from an offline video that requires format conversion.

| | Preprocessing | [qtimlvconverter](https://docs.qualcomm.com/doc/80-80021-50/topic/qtimlvconverter.html) does the following to prepare the video stream for inferencing:

Color conversion

Resize

Mean subtraction

| | Inferencing |

The preprocessed data is sent to the inference plugin in a tensor format.

The inference plugin sends this data to the machine learning engines:

LiteRT engine uses qtimltflite.

Qualcomm Neural Processing SDK engine uses qtimlsnpe.

| | Postprocessing | The output tensor is passed down to the postprocessing plugin [qtimlpostprocess](https://docs.qualcomm.com/doc/80-80021-50/topic/qtimlpostprocess.html). | | Overlay | The overlay plugin that interprets the machine learning metadata.

qtivoverlay draws appropriate overlay on the buffer that allows the inference result to be visualized in real-time.

The stream can be rendered on to the display, streamed over network, or encoded and stored to a local storage.

| | Sink |

Waylandsink is used to display the output on a screen.

Filesink is used to save the inferenced video to a local storage.

| ### Related information [Run machine learning use cases](https://docs.qualcomm.com/doc/80-80021-50/topic/machine-learning-use-cases.html) Last Published: Mar 26, 2026 [Previous Topic Qualcomm GStreamer architecture and plugins](https://docs.qualcomm.com/bundle/publicresource/80-80021-50/topics/gst-plugin-architecture.md) [Next Topic Configure Qualcomm GStreamer plugins](https://docs.qualcomm.com/bundle/publicresource/80-80021-50/topics/qim-sdk-plugins.md)