# GStreamer plugin architecture
Source: [https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html)
The Qualcomm IM SDK hides the complexity of the hardware within the plugin
architecture and provides APIs to applications. Using this framework, you can create
applications without the need to access the low-level platform libraries and hardware
details, which can vary across platforms.
The [Configure Qualcomm GStreamer plugins](https://docs.qualcomm.com/doc/80-70018-50/topic/qim-sdk-plugins.html) include video encoding/decoding, camera ISP, GPU,
display, audio DSP (aDSP), and AI/ML accelerators.
Figure : Qualcomm IM SDK GStreamer plugin architecture

## Graphics and display architecture
Source: [https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html)
The Wayland server uses the Wayland protocol to enable communication between the
display and graphics servers, and its clients. The display and graphics architecture uses
the Weston server (Weston).
Weston manages both composition and display while it runs as a separate process in the
system.
Figure : Weston/Wayland architecture

Table : Weston/Wayland components
| Component | Description |
| --- | --- |
| Wayland/ GLES client | Implements the Wayland protocol for Weston. |
| Weston server | Implements the Wayland compositor.
- Uses the Kernel Mode Setting (KMS) to set up the display.
- Performs the compositing using OpenGL ES and Direct Rendering
Manager (DRM).
|
| Secure dynamic messaging (SDM) back-end |
- Uses display hardware abstraction layer (HAL) to interact with
the display hardware.
- Provides several platform-dependent implementations with one of
it designated for DRM and KMS.
|
| GBM buffer management library (libGBM) |
- Includes DMA back-end.
- Used for zero-copy buffer sharing between display and
graphics.
|
| EGL platform-specific driver (EGL sub-driver) | Interacts with GBM and Wayland protocol to communicate with the
Weston compositor. |
## Camera architecture
Source: [https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html)
The camera architecture provides a high-level abstraction on how the underlying
camera driver and hardware interact with the [qtiqmmfsrc](https://docs.qualcomm.com/doc/80-70018-50/topic/qtiqmmfsrc.html) camera
plugin.
The figure shows the camera architecture, where the qtimmfsrc plugin, acts as a client to
the Qualcomm Camera Service.
Figure : Camera pipeline

Table : Camera pipeline components
| Component | Description |
| --- | --- |
| Linux embedded (LE) camera service |
- Runs as a daemon in the system:
- Provides easy remote procedure call (RPC) APIs to
control the camera.
- Exposes helper client APIs, which perform RPC between
client and server.
- Uses the HAL3 API, which interacts with the camera back-end
(CamX) and camera driver to configure camera sensor and image
signal processor (ISP) hardware.
- Uses GBM to assign buffers for each camera stream and submits
them to HAL3.
|
| Client/server architecture | Allows the multi-client and multi-camera use cases to do the
following:
- Create many instances of qtiqmmfsrc where one instance of the
plugin corresponds to one physical or logical camera.
- Implement camera use cases. These instances can be in the same
process or different processes depending upon the use case
|
| qtiqmmfsrc | |
| [Waylandsink](https://docs.qualcomm.com/doc/80-70018-50/topic/waylandsink.html) |
- Uses the raw YUV (NV12/21) stream output to render the camera
frames to a physical display.
- Achieves a live camera preview use case.
- Allows the postprocessing element to improve the quality.
- Allows the ML inferencing elements to do inferencing on a live
camera.
|
| Buffers |
- The same buffers are circulated to the clients (qtiqmmfsrc) and
other plugins in the pipeline.
- The buffer pools manage the buffers to avoid per-frame
allocation.
|
**Related Resources**
- [Camera use cases](https://docs.qualcomm.com/doc/80-70018-50/topic/camera.html)
## Video architecture
Source: [https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html)
The [v4l2h264enc](https://docs.qualcomm.com/doc/80-70018-50/topic/v4l2h264enc.html) and [v4l2h265enc](https://docs.qualcomm.com/doc/80-70018-50/topic/v4l2h265enc.html) video encode
plugins connect with the camera plugin to implement the advance video encode (H.264 or
H.265) use cases. The [v4l2h264dec](https://docs.qualcomm.com/doc/80-70018-50/topic/v4l2h264dec.html) and [v4l2h265dec](https://docs.qualcomm.com/doc/80-70018-50/topic/v4l2h265dec.html)
video decode plugins connect with Waylandsink to implement the video playback use
cases.
### Encode
The figure shows the architecture for video encoding. The v4l2h264enc and v4l2h265enc
plugins are connected to a file multiplexer (MP4 or MPEGTS) to save the encode
bitstream to the file system.
For video encode use cases, see [Video encode and decode](https://docs.qualcomm.com/doc/80-70018-50/topic/camera-and-video-encode.html).
Figure : Video encode pipeline

Table : Video encode components
| Components | Description |
| --- | --- |
| qtiqmmfsrc | Captures and provides several video streams in parallel. For more
information about the camera service, see [Camera architecture](https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html#camera). |
| Video encode plugins | Uses the V4L2 driver and APIs to encode the video stream.
- v4l2h264enc: Encodes the video for AVC
(H.264) format.
- v4l2h265enc: Encodes the video for HEVC
(H.265) format.
|
| H.264/H.265 parse | Parses the video stream. |
| Filemux | Multiplexes encoded video stream and saves to a file. |
### Decode
The figure shows the architecture for video decoding where the video stream from a
file is demultiplexed, decoded, and displayed on a local device.
The decode parameters are exposed as a property to the application to provide control
over the video decode pipeline.
Figure : Video decode pipeline

For video decode use cases, see [Video playback use cases](https://docs.qualcomm.com/doc/80-70018-50/topic/video-playback-use-cases.html).
Table : Video decode components
| Components | Description |
| --- | --- |
| Video decode plugins | Uses the V4L2 driver and APIs to decode the video stream and
connects with the Waylandsink plugin.
- v4l2h264dec: Decodes the video for AVC
(H.264) format.
- v4l2h265dec: Decodes the video for HEVC
(H.265) format.
|
| Waylandsink |
- Receives the GBM buffers (decoded buffers) as an
input.
- Sends them to the Weston server (through Wayland protocol)
for display composition.
|
## Audio architecture
Source: [https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html)
For audio capture and playback, use the [pulsesrc](https://docs.qualcomm.com/doc/80-70018-50/topic/pulsesrc.html) and [pulsesink](https://docs.qualcomm.com/doc/80-70018-50/topic/pulsesink.html) GStreamer plugins. You can encode and decode audio through an
open-source software.
### Capture
The figure shows the audio capture using the pulsesrc plugin. The audio stream is
then processed and written to a file.
Figure : Audio capture pipeline

Table : Audio capture components
| Component | Description |
| --- | --- |
| Pulsesrc |
- Captures the audio and shares it with the underlying
PulseAudio server.
- Sets the audio source from which the pulse-code modulation
(PCM) audio samples are captured.
- Saves the PCM samples to a file.
|
| PulseAudio server |
- Interacts with the Qualcomm hardware.
- Uses a pluggable module to interact with the ALSA driver to
process the audio data.
|
### Playback
Use the pulsesink plugin for audio playback. It's an upstream plugin that allows you
to play audio from various audio sources (live source or an encoded audio file).
Figure : Audio playback pipeline

Table : Audio playback components
| Component | Description |
| --- | --- |
| Pulsesink |
- Interacts with the underlying PulseAudio server.
- Sets the audio playback sink on which the audio is
played.
- Plays only the PCM data.
|
| PulseAudio server |
- Interacts with the Qualcomm hardware.
- Uses a pluggable module to interact with the ALSA driver to
process the audio data for playback.
|
### Encode
You can encode the audio using an open-source audio encoder plugin.
Figure : Audio encoding pipeline

Table : Audio encoding components
| Component | Description |
| --- | --- |
| Pulsesrc | Captures the audio and shares it with the underlying PulseAudio
server. |
| PulseAudio server | Interacts with the ALSA driver to process the audio data for
encoding. |
| Encode | Encodes the audio data using an open-source software. |
| Parse | Parses the audio data. |
| Filemux | Multiplexes it and saves it in an MP4 or MPEGTS
container. |
### Decode
You can decode the audio using an open-source audio decoder plugin.
Figure : Audio decode pipeline

Table : Audio decode components
| Component | Description |
| --- | --- |
| Filesrc | Reads the audio data. |
| Stream demux | Demultiplexes the audio data. |
| Decode | Decodes the audio data using an open-source software. |
| Pulsesink | Interacts with the underlying PulseAudio server and plays the
decoded audio. |
| PulseAudio server | Interacts with the ALSA driver to process the audio data for
decoding. |
## Machine learning architecture
Source: [https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html](https://docs.qualcomm.com/doc/80-70018-50/topic/architecture.html)
Qualcomm IM SDK supports machine learning use cases that include video preprocessing,
model inference, output tensor postprocessing, and inference result overlay to
stream.
The machine learning framework provides the following types of video analytics:
- Image classification
- Object detection
- Image segmentation
With machine learning plugins in the framework, use the following for inferencing:
- [LiteRT](https://ai.google.dev/edge/litert) (or TFLite)
- [Qualcomm Neural Processing SDK](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-2/overview.html)
For each of the engines, the delegates speed up the model inference performance.
### ML pipeline processing
The figure depicts
the pipeline in which the captured video/audio stream is processed among the
preprocessing, inference, and postprocessing plugins.
Figure : ML pipeline

The table lists the sequential stages during which the video/audio stream is
captured, processed, and the output is either displayed on a screen or saved to a
file.
Table : ML pipeline processing stages
| Process | Description |
| --- | --- |
| Source |
- qtiqmmfsrc captures the YUV streams from
live camera source.
- The filesrc plugin captures any format from an offline video
that requires format conversion.
|
| Preprocessing | [qtimlvconverter](https://docs.qualcomm.com/doc/80-70018-50/topic/qtimlvconverter.html)does the following to prepare
the video stream for inferencing:
- Color conversion
- Resize
- Mean subtraction
|
| Inferencing |
- The preprocessed data is sent to the inference plugin in a
tensor format.
- The inference plugin sends this data to the machine learning
engines:
|
| Postprocessing | The output tensor is passed down to the postprocessing
plugins: |
| Overlay | The overlay plugin that interprets the machine learning
metadata.
- qtivoverlay draws appropriate overlay on
the buffer that allows the inference result to be visualized
in real-time.
- The stream can be rendered on to the display, streamed over
network, or encoded and stored to a local storage.
|
| Sink |
- Waylandsink is used to display the
output on a screen.
- Filesink is used to save the inferenced video to a local
storage.
|
**Related Resources**
- [Run machine learning use cases](https://docs.qualcomm.com/doc/80-70018-50/topic/machine-learning-use-cases.html)
Last Published: Jan 30, 2026
[Previous Topic
Qualcomm GStreamer architecture and plugins](https://docs.qualcomm.com/bundle/publicresource/80-70018-50/topics/gst-plugin-architecture.md) [Next Topic
Configure Qualcomm GStreamer plugins](https://docs.qualcomm.com/bundle/publicresource/80-70018-50/topics/qim-sdk-plugins.md)