# GStreamer plugin architecture
Source: [https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html)
The GStreamer plugins interact with Qualcomm IM SDK multimedia components to build
the multimedia or AI/ML use cases.
The [Qualcomm GStreamer plugins](https://docs.qualcomm.com/doc/80-70015-50/topic/qim-sdk-plugins.html) include video encoding/decoding, camera ISP, GPU,
display, audio DSP (aDSP), and AI/ML accelerators.
The Qualcomm IM SDK hides the complexity of the hardware within the plugin architecture
and provides APIs to applications. Using this framework, you can create applications
without the need to access the low-level platform libraries and hardware details, which
can vary across platforms.
Figure : Qualcomm IM SDK GStreamer plugin architecture

## Graphics and display
Source: [https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html)
The Wayland server uses the Wayland protocol to enable communication between the
display and graphics servers, and its clients. The display and graphics architecture uses
the Weston server (Weston).
Weston is dedicated to the management of composition and display with the capability to
run as a separate process in the system.
Figure : Weston/Wayland architecture

Table : Weston/Wayland components
| Component | Description |
| --- | --- |
| Wayland/ GLES client | Implements the Wayland protocol for Weston. |
| Weston server | Implements the Wayland compositor.
- Uses the Kernel Mode Setting (KMS) to set up the display.
- Performs the compositing using OpenGL ES and Direct Rendering
Manager (DRM)
|
| Secure dynamic messaging (SDM) back-end |
- Uses display hardware abstraction layer (HAL) to interact with
the display hardware.
- Provides multiple platform-dependent implementations with one of
it designated for DRM and KMS.
|
| GBM buffer management library (libGBM) |
- Includes DMA back-end.
- Used for zero-copy buffer sharing between display and
graphics.
|
| EGL platform-specific driver (EGL sub-driver) | Interacts with GBM and Wayland protocol to communicate with the
Weston compositor. |
## Camera
Source: [https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html)
The camera architecture provides a high-level abstraction on how the underlying
camera driver and hardware interact with the [qtiqmmfsrc](https://docs.qualcomm.com/doc/80-70015-50/topic/qtiqmmfsrc.html) camera
plugin.
The figure shows the camera architecture, where the qtimmfsrc plugin, acts as a client to
the Qualcomm Camera Service.
Figure : Camera pipeline

Table : Camera pipeline components
| Component | Description |
| --- | --- |
| Linux embedded (LE) camera service |
- Runs as a daemon in the system:
- Provides easy remote procedure call (RPC) APIs to
control the camera.
- Exposes helper client APIs, which perform RPC between
client and server.
- Uses the HAL3 API, which interacts with the camera back-end
(CamX) and camera driver to configure camera sensor and image
signal processor (ISP) hardware.
- Uses GBM to allocate buffers for each camera stream and submits
them to HAL3.
|
| Client-server architecture | Enables the multi-client and multi-camera use cases to do the
following:
- Create many instances of qtiqmmfsrc where one instance of the
plugin corresponds to one physical or logical camera.
- Implement camera use cases. These instances can be in the same
process or different processes depending upon the use case
|
| qtiqmmfsrc | |
| [Waylandsink](https://docs.qualcomm.com/doc/80-70015-50/topic/waylandsink.html) |
- Uses the raw YUV (NV12/21) stream output to render the camera
frames to a physical display.
- Achieves a live camera preview use case.
- Enables the postprocessing element to improve the quality.
- Enables the ML inferencing elements to do inferencing on a live
camera.
|
| Buffers |
- The same buffers are circulated to the clients (qtiqmmfsrc) and
other plugins in the pipeline.
- The buffer pools manage the buffers to avoid per-frame
allocation.
|
**Related Resources**
- [Camera use cases](https://docs.qualcomm.com/doc/80-70015-50/topic/camera.html)
## Video
Source: [https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html)
The [v4l2h264enc](https://docs.qualcomm.com/doc/80-70015-50/topic/v4l2h264enc.html) and [v4l2h265enc](https://docs.qualcomm.com/doc/80-70015-50/topic/v4l2h265enc.html) video encode
plugins connect with the camera plugin to implement the advance video encode (H.264 or
H.265) use cases. The [v4l2h264dec](https://docs.qualcomm.com/doc/80-70015-50/topic/v4l2h264dec.html) and [v4l2h265dec](https://docs.qualcomm.com/doc/80-70015-50/topic/v4l2h265dec.html)
video decode plugins connect with Waylandsink to implement the video playback use
cases.
### Encode
The figure shows the architecture for video encoding. The v4l2h264enc and v4l2h265enc
plugins can be connected to a file multiplexer (MP4 or MPEGTS) to save the encode
bitstream to the file system.
For video encode use cases, see [Camera and video encode](https://docs.qualcomm.com/doc/80-70015-50/topic/camera-and-video-encode.html).
Figure : Video encode pipeline

Table : Video encode components
| Components | Description |
| --- | --- |
| qtiqmmfsrc | Captures and provides multiple video streams in parallel. For
more information on the camera service, see [Camera](https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html#camera). |
| Video encode plugins | Uses the V4L2 driver and APIs to encode the video stream.
- v4l2h264enc: Encodes the video for AVC
(H.264) format.
- v4l2h265enc: Encodes the video for HEVC
(H.265) format.
|
| H.264/H.265 parse | Parses the video stream. |
| Filemux | Multiplexes encoded video stream and saves to a file. |
### Decode
The figure shows the architecture for video decoding where the video stream from a
file is demultiplexed, decoded, and displayed on a local device.
The decode parameters are exposed as a property to the application to provide control
over the video decode pipeline.
Figure : Video decode pipeline

For video decode use cases, see [Video playback use cases](https://docs.qualcomm.com/doc/80-70015-50/topic/video-playback-use-cases.html).
Table : Video decode components
| Components | Description |
| --- | --- |
| Video decode plugins | Uses the V4L2 driver and APIs to decode the video stream and
connects with the Waylandsink plugin.
- v4l2h264dec: Decodes the video for AVC
(H.264) format.
- v4l2h265dec: Decodes the video for HEVC
(H.265) format.
|
| Waylandsink |
- Receives the GBM buffers (decoded buffers) as an
input.
- Sends them to the Weston server (via Wayland protocol) for
display composition.
|
## Audio
Source: [https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html)
For audio capture and playback, the [pulsesrc](https://docs.qualcomm.com/doc/80-70015-50/topic/pulsesrc.html) and [pulsesink](https://docs.qualcomm.com/doc/80-70015-50/topic/pulsesink.html) GStreamer plugins are used. Audio encoding and decoding are done
through an open-source software.
### Capture
The figure shows the audio being captured using the pulsesrc plugin, which is
processed and written to a file.
Figure : Audio capture pipeline

Table : Audio capture components
| Component | Description |
| --- | --- |
| Pulsesrc |
- Captures the audio and shares it with the underlying
PulseAudio server.
- Sets the audio source from which the pulse-code modulation
(PCM) audio samples are captured.
- Saves the PCM samples to a file.
|
| PulseAudio server |
- Interacts with the Qualcomm hardware.
- Uses a pluggable module to interact with the ALSA driver to
process the audio data.
|
### Playback
The pulsesink plugin is used for audio playback. It is an upstream plugin that
enables you to play audio from various audio sources (live source or an encoded
audio file).
Figure : Audio playback pipeline

Table : Audio playback components
| Component | Description |
| --- | --- |
| Pulsesink |
- Interacts with the underlying PulseAudio server.
- Sets the audio playback sink on which the audio has to be
played.
- Plays only the PCM data.
|
| PulseAudio server |
- Interacts with the Qualcomm hardware.
- Uses a pluggable module to interact with the ALSA driver to
process the audio data for playback.
|
### Encode
Audio encoding is done using an open-source audio encoder plugin.
Figure : Audio encoding pipeline

Table : Audio encoding components
| Component | Description |
| --- | --- |
| Pulsesrc | Captures the audio and shares it with the underlying PulseAudio
server. |
| PulseAudio server | Interacts with the ALSA driver to process the audio data for
encoding. |
| Encode | Encodes the audio data using an open-source software. |
| Parse | Parses the audio data. |
| Filemux | Multiplexes it and saves it in an MP4 or MPEGTS
container. |
### Decode
Audio decoding is done using an open-source audio decoder plugin.
Figure : Audio decode pipeline

Table : Audio decode components
| Component | Description |
| --- | --- |
| Filesrc | Reads the audio data. |
| Stream demux | Demultiplexes the audio data. |
| Decode | Decodes the audio data using an open-source software. |
| Pulsesink | Interacts with the underlying PulseAudio server and plays the
decoded audio. |
| PulseAudio server | Interacts with the ALSA driver to process the audio data for
decoding. |
## Machine learning
Source: [https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70015-50/topic/architecture.html)
Qualcomm IM SDK supports machine learning use cases that include video
pre-processing, model inference, output tensor post-processing, and inference result overlay
to stream.
The machine learning framework provides the following types of video analytics:
- Image classification
- Object detection
- Image segmentation
With machine learning plugins in the framework, use the following for inferencing:
- [TensorFlow Lite (TFLite)](https://www.tensorflow.org/lite/guide)
- [Qualcomm® Neural Processing SDK](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-2/overview.html) (formerly
known as Qualcomm Snapdragon Neural Processing Engine (SNPE))
For each of the engines, delegates are enabled to accelerate the model inference
performance.
### ML pipeline processing
The figure depicts
the pipeline in which the captured video/audio stream is processed among the
preprocessing, inference, and post-processing plugins.
Figure : ML pipeline

The table lists the sequential stages during which the video/audio stream is
captured, processed, and the output is either displayed on a screen or saved to a
file.
Table : ML pipeline processing stages
| Process | Description |
| --- | --- |
| Source |
- qtiqmmfsrc captures the YUV streams from
live camera source.
- The filesrc plugin captures any format from an offline video
that requires format conversion.
|
| Preprocessing | [qtimlvconverter](https://docs.qualcomm.com/doc/80-70015-50/topic/qtimlvconverter.html)does the following to prepare
the video stream for inferencing:
- Color conversion
- Resize
- Mean subtraction
|
| Inferencing |
- The preprocessed data is sent to the inference plugin in a
tensor format.
- The inference plugin sends this data to the machine learning
engines:
|
| Postprocessing | The output tensor gets passed down to the postprocessing
plugins: |
| Overlay | The overlay plugin that interprets the machine learning
metadata.
- qtioverlay draws appropriate overlay on
the buffer that allows the inference result to be visualized
in real-time.
- The stream can be rendered on to the display, streamed over
network, or encoded and stored to a local storage.
|
| Sink |
- Waylandsink is used to display the
output on a screen.
- Filesink is used to save the inferenced video to a local
storage.
|
**Related Resources**
- [Machine learning use cases](https://docs.qualcomm.com/doc/80-70015-50/topic/machine-learning-use-cases.html)
Last Published: Oct 27, 2025
[Previous Topic
Single camera stream and socket use case](https://docs.qualcomm.com/bundle/publicresource/80-70015-50/topics/socketsrc-and-socketsink-use-case.md) [Next Topic
Qualcomm GStreamer plugins](https://docs.qualcomm.com/bundle/publicresource/80-70015-50/topics/qim-sdk-plugins.md)