# GStreamer plugin architecture Source: [https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html) The Qualcomm IM SDK hides the complexity of the hardware within the plugin architecture and provides APIs to applications. Using this framework, you can create applications without the need to access the low-level platform libraries and hardware details, which can vary across platforms. The [Configure Qualcomm GStreamer plugins](https://docs.qualcomm.com/doc/80-70020-50/topic/qim-sdk-plugins.html) include video encoding/decoding, camera ISP, GPU, display, audio DSP (aDSP), and AI/ML accelerators. Figure : Qualcomm IM SDK GStreamer plugin architecture ## Graphics and display architecture Source: [https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html) The Wayland server uses the Wayland protocol to enable communication between the display and graphics servers, and its clients. The display and graphics architecture uses the Weston server (Weston). Weston manages both composition and display while it runs as a separate process in the system. Figure : Weston/Wayland architecture Table : Weston/Wayland components | Component | Description | | --- | --- | | Wayland/ GLES client | Implements the Wayland protocol for Weston. | | Weston server | Implements the Wayland compositor.

Uses the Kernel Mode Setting (KMS) to set up the display.

Performs the compositing using OpenGL ES and Direct Rendering
Manager (DRM).

| | Secure dynamic messaging (SDM) back-end |

Uses display hardware abstraction layer (HAL) to interact with
the display hardware.

Provides several platform-dependent implementations with one of
it designated for DRM and KMS.

| | GBM buffer management library (libGBM) |

Includes DMA back-end.

Used for zero-copy buffer sharing between display and
graphics.

| | EGL platform-specific driver (EGL sub-driver) | Interacts with GBM and Wayland protocol to communicate with the
Weston compositor. | ## Camera architecture Source: [https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html) The camera architecture provides a high-level abstraction on how the underlying camera driver and hardware interact with the [qtiqmmfsrc](https://docs.qualcomm.com/doc/80-70020-50/topic/qtiqmmfsrc.html) camera plugin. The following figure shows the camera architecture, where the qtimmfsrc plugin, acts as a client to the Qualcomm Camera Service. Figure : Camera pipeline Table : Camera pipeline components | Component | Description | | --- | --- | | Linux embedded (LE) camera service |

Runs as a daemon in the system:
- Provides easy remote procedure call (RPC) APIs to
  control the camera.
- Exposes helper client APIs, which perform RPC between
  client and server.

Uses the HAL3 API, which interacts with the camera back-end
(CamX) and camera driver to configure camera sensor and image
signal processor (ISP) hardware.

Uses GBM to assign buffers for each camera stream and submits
them to HAL3.

| | Client/server architecture | Allows the multi-client and multi-camera use cases to do the
following:

Create many instances of qtiqmmfsrc where one instance of the
plugin corresponds to one physical or logical camera.

Implement camera use cases. These instances can be in the same
process or different processes depending upon the use case

| | qtiqmmfsrc |

Provides several streams in parallel (many source pads), and
each stream can be of different formats (NV12/21 or MJPEG).

For example, if there are three parallel streams, they're
processed as follows:
1. One stream for local storage.
2. One stream for YUV stream for live camera preview on a
  local display.
3. One for network streaming.

Supports ‘N’ number of parallel streams, where 'N' depends on
camera ISP capability.

| | [Waylandsink](https://docs.qualcomm.com/doc/80-70020-50/topic/waylandsink.html) |

Uses the raw YUV (NV12/21) stream output to render the camera
frames to a physical display.

Achieves a live camera preview use case.

Allows the postprocessing element to improve the quality.

Allows the ML inferencing elements to do inferencing on a live
camera.

| | Buffers |

The same buffers are circulated to the clients (qtiqmmfsrc) and
other plugins in the pipeline.

The buffer pools manage the buffers to avoid per-frame
allocation.

| ### Related information [Camera use cases](https://docs.qualcomm.com/doc/80-70020-50/topic/camera.html) ## Video architecture Source: [https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html) The [v4l2h264enc](https://docs.qualcomm.com/doc/80-70020-50/topic/v4l2h264enc.html) and [v4l2h265enc](https://docs.qualcomm.com/doc/80-70020-50/topic/v4l2h265enc.html) video encode plugins connect with the camera plugin to implement the advance video encode (H.264 or H.265) use cases. The [v4l2h264dec](https://docs.qualcomm.com/doc/80-70020-50/topic/v4l2h264dec.html) and [v4l2h265dec](https://docs.qualcomm.com/doc/80-70020-50/topic/v4l2h265dec.html) video decode plugins connect with Waylandsink to implement the video playback use cases. ### Encode The following figure shows the architecture for video encoding. The v4l2h264enc and v4l2h265enc plugins are connected to a file multiplexer (MP4 or MPEGTS) to save the encode bitstream to the file system. For video encode use cases, see [Video encode and decode](https://docs.qualcomm.com/doc/80-70020-50/topic/camera-and-video-encode.html). Figure : Video encode pipeline Table : Video encode components | Components | Description | | --- | --- | | qtiqmmfsrc | Captures and provides several video streams in parallel. For more
information about the camera service, see [Camera architecture](https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html#camera). | | Video encode plugins | Uses the V4L2 driver and APIs to encode the video stream.

v4l2h264enc: Encodes the video for AVC
(H.264) format.

v4l2h265enc: Encodes the video for HEVC
(H.265) format.

| | H.264/H.265 parse | Parses the video stream. | | Filemux | Multiplexes encoded video stream and saves to a file. | ### Decode The following figure shows the architecture for video decoding where the video stream from a file is demultiplexed, decoded, and displayed on a local device. The decode parameters are exposed as a property to the application to provide control over the video decode pipeline. Figure : Video decode pipeline For video decode use cases, see [Video playback use cases](https://docs.qualcomm.com/doc/80-70020-50/topic/video-playback-use-cases.html). Table : Video decode components | Components | Description | | --- | --- | | Video decode plugins | Uses the V4L2 driver and APIs to decode the video stream and
connects with the Waylandsink plugin.

v4l2h264dec: Decodes the video for AVC
(H.264) format.

v4l2h265dec: Decodes the video for HEVC
(H.265) format.

| | Waylandsink |

Receives the GBM buffers (decoded buffers) as an
input.

Sends them to the Weston server (through Wayland protocol)
for display composition.

| ## Audio architecture Source: [https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html) For audio capture and playback, use the [pulsesrc](https://docs.qualcomm.com/doc/80-70020-50/topic/pulsesrc.html) and [pulsesink](https://docs.qualcomm.com/doc/80-70020-50/topic/pulsesink.html) GStreamer plugins. You can encode and decode audio through an open-source software. ### Capture The following figure shows the audio capture using the pulsesrc plugin. The audio stream is then processed and written to a file. Figure : Audio capture pipeline Table : Audio capture components | Component | Description | | --- | --- | | Pulsesrc |

Captures the audio and shares it with the underlying
PulseAudio server.

Sets the audio source from which the pulse-code modulation
(PCM) audio samples are captured.

Saves the PCM samples to a file.

| | PulseAudio server |

Interacts with the Qualcomm hardware.

Uses a pluggable module to interact with the ALSA driver to
process the audio data.

| ### Playback Use the pulsesink plugin for audio playback. It's an upstream plugin that allows you to play audio from various audio sources (live source or an encoded audio file). Figure : Audio playback pipeline Table : Audio playback components | Component | Description | | --- | --- | | Pulsesink |

Interacts with the underlying PulseAudio server.

Sets the audio playback sink on which the audio is
played.

Plays only the PCM data.

| | PulseAudio server |

Interacts with the Qualcomm hardware.

Uses a pluggable module to interact with the ALSA driver to
process the audio data for playback.

| ### Encode You can encode the audio using an open-source audio encoder plugin. Figure : Audio encoding pipeline Table : Audio encoding components | Component | Description | | --- | --- | | Pulsesrc | Captures the audio and shares it with the underlying PulseAudio
server. | | PulseAudio server | Interacts with the ALSA driver to process the audio data for
encoding. | | Encode | Encodes the audio data using an open-source software. | | Parse | Parses the audio data. | | Filemux | Multiplexes it and saves it in an MP4 or MPEGTS
container. | ### Decode You can decode the audio using an open-source audio decoder plugin. Figure : Audio decode pipeline Table : Audio decode components | Component | Description | | --- | --- | | Filesrc | Reads the audio data. | | Stream demux | Demultiplexes the audio data. | | Decode | Decodes the audio data using an open-source software. | | Pulsesink | Interacts with the underlying PulseAudio server and plays the
decoded audio. | | PulseAudio server | Interacts with the ALSA driver to process the audio data for
decoding. | ## Machine learning architecture Source: [https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70020-50/topic/architecture.html) Qualcomm IM SDK supports machine learning use cases that include video preprocessing, model inference, output tensor postprocessing, and inference result overlay to stream. The machine learning framework provides the following types of video analytics: - Image classification - Object detection - Image segmentation With machine learning plugins in the framework, use the following for inferencing: - [LiteRT](https://ai.google.dev/edge/litert) (or TFLite) - [Qualcomm Neural Processing SDK](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-2/overview.html) For each of the engines, the delegates speed up the model inference performance. ### ML pipeline processing The following figure depicts the pipeline in which the captured video/audio stream is processed among the preprocessing, inference, and postprocessing plugins. Figure : ML pipeline The following table lists the sequential stages during which the video/audio stream is captured, processed, and the output is either displayed on a screen or saved to a file. Table : ML pipeline processing stages | Process | Description | | --- | --- | | Source |

qtiqmmfsrc captures the YUV streams from
live camera source.

The filesrc plugin captures any format from an offline video
that requires format conversion.

| | Preprocessing | [qtimlvconverter](https://docs.qualcomm.com/doc/80-70020-50/topic/qtimlvconverter.html)does the following to prepare
the video stream for inferencing:

Color conversion

Resize

Mean subtraction

| | Inferencing |

The preprocessed data is sent to the inference plugin in a
tensor format.

The inference plugin sends this data to the machine learning
engines:
- LiteRT engine uses qtimltflite.
- Qualcomm Neural Processing SDK engine uses qtimlsnpe.

| | Postprocessing | The output tensor is passed down to the postprocessing
plugins:

Object detection–qtimlvdetection

Image classification–qtimlvclassification

Image segmentation–qtimlvsegmentation

| | Overlay | The overlay plugin that interprets the machine learning
metadata.

qtivoverlay draws appropriate overlay on
the buffer that allows the inference result to be visualized
in real-time.

The stream can be rendered on to the display, streamed over
network, or encoded and stored to a local storage.

| | Sink |

Waylandsink is used to display the
output on a screen.

Filesink is used to save the inferenced video to a local
storage.

| ### Related information [Run machine learning use cases](https://docs.qualcomm.com/doc/80-70020-50/topic/machine-learning-use-cases.html) Last Published: Jan 30, 2026 [Previous Topic Qualcomm GStreamer architecture and plugins](https://docs.qualcomm.com/bundle/publicresource/80-70020-50/topics/gst-plugin-architecture.md) [Next Topic Configure Qualcomm GStreamer plugins](https://docs.qualcomm.com/bundle/publicresource/80-70020-50/topics/qim-sdk-plugins.md)