# Development process for new plugin This section explains how to develop a new super resolution (qtimlvsuperresolution) plugin. - The eSDK needs to be setup to download and compile application/plugin code. See [Compile application: Qualcomm IM SDK](https://docs.qualcomm.com/doc/80-70017-15B/topic/compile-application-qualcomm-im-sdk.html) for instructions on setting up the eSDK and downloading and compiling plugin code. - Use the source code of the existing plugins as a reference throughout this process. [CodeLinaro/ le / platform / vendor / qcom-opensource / gst-plugins-qti-oss · GitLab](https://git.codelinaro.org/clo/le/platform/vendor/qcom-opensource/gst-plugins-qti-oss/-/tree/imsdk.lnx.2.0.0.r1-rel?ref_type=heads) ## Plugin Directory Structure gst-plugins-qti-oss/gst-plugins-mlv/ CMakeLists.txt config.h.in mlv.c mlv.h modules/ CMakeLists.txt ml-video--module.h ml-v-.c Copy to clipboard ## Steps to create superresolution plugin 1. Create the plugin header (`gst-plugins-mlvsuperresolution/mlvsuperresolution.h`). Use the existing plugin as a reference: [gst-plugin-mlvsegmentation/mlvsegmentation.h·imsdk.lnx.2.0.0.r2-rel · CodeLinaro / le / platform / vendor / qcom-opensource / gst-plugins-qti-oss · GitLab](https://git.codelinaro.org/clo/le/platform/vendor/qcom-opensource/gst-plugins-qti-oss/-/blob/imsdk.lnx.2.0.0.r2-rel/gst-plugin-mlvsegmentation/mlvsegmentation.h?ref_type=heads) 1. Import the required headers. 2. Create a structure with the data variables required for processing. Example super resolution inheritance chain: GObject > GstObject > GstElement > GstBaseTransform GstBaseTransform is suitable for elements where the output size and capabilities are entirely determined by the input size and capabilities struct _GstMLVideoSuperresolution {GstBaseTransform parent; GstMLInfo *mlinfo; GstVideoInfo *vinfo; GstBufferPool *outpool; GstMLModule *module; gint mdlenum; GstStructure *mlconstants; }; struct _GstMLVideoSuperresolutionClass {GstBaseTransformClass parent; }; Copy to clipboard Note The labels variable is not required for superresolution. Make sure to remove the labels variable and its usage in code. 2. Create the plugin implementation file (`gst-plugins-mlvsuperresolution/mlvsuperresolution.c`). Use the existing plugin as a reference: [gst-plugin-mlvsegmentation/mlvsegmentation.c imsdk.lnx.2.0.0.r2-rel · CodeLinaro / le / platform / vendor / qcom-opensource / gst-plugins-qti-oss · GitLab](https://git.codelinaro.org/clo/le/platform/vendor/qcom-opensource/gst-plugins-qti-oss/-/blob/imsdk.lnx.2.0.0.r2-rel/gst-plugin-mlvsegmentation/mlvsegmentation.c?ref_type=heads) 1. Import the `mlvsuperresolution.h` and other required headers. 2. Define the caps - video formats #define GST_ML_VIDEO_SUPERRESOLUTION_VIDEO_FORMATS "{ RGBA, BGRA, ARGB, ABGR, RGBx, BGRx, xRGB, xBGR, RGB, BGR }" Copy to clipboard - src caps #define GST_ML_VIDEO_SUPER_RESOLUTION_SRC_CAPS \ "video/x-raw, " \ "format = (string) " GST_ML_VIDEO_SUPER_RESOLUTION_VIDEO_FORMATS "; " \ "video/x-raw(" GST_CAPS_FEATURE_MEMORY_GBM "), " \ "format = (string) " GST_ML_VIDEO_SUPER_RESOLUTION_VIDEO_FORMATS Copy to clipboard - sink caps #define GST_ML_VIDEO_SUPER_RESOLUTION_SINK_CAPS \ "neural-network/tensors" Copy to clipboard 3. Implement following functions - Init 1. Register a custom plugin. Once registered you can access plugin in reference apps with `gst_element_factory_make("plugin-name","plugin-name");` static gboolean plugin_init(GstPlugin * plugin); Copy to clipboard 2. Initialize superresolution with default parameters. This allows you to control output buffers. On receiving an input buffer, `gst_ml_video_super_resolution_prepare_output_buffer` is called to allocate an output buffer. static void gst_ml_video_super_resolution_init(GstMLVideoSuperResolution * super_resolution) Copy to clipboard 3. Initialize callback functions to the function pointer, sets caps, and default properties. static void gst_ml_video_super_resolution_class_init(GstMLVideoSuperResolutionClass * klass) Copy to clipboard - Buffer management 1. Allocates either graphics Buffer Manager (GBM) or ION Memory depending on the negotiated caps. static GstBufferPool * gst_ml_video_super_resolution_create_pool (GstMLVideoSuperResolution * super_resolution, GstCaps * caps) Copy to clipboard 2. Ensures appropriate buffer pool configuration based on the caps. gst_ml_video_super_resolution_decide_allocation (GstBaseTransform * base, GstQuery * query) Copy to clipboard 3. Ensures output buffers are allocated from the pool depending on caps. static GstFlowReturn gst_ml_video_super_resolution_prepare_output_buffer (GstBaseTransform * base, GstBuffer * inbuffer, GstBuffer ** outbuffer) Copy to clipboard - Caps 1. Validate the module and then collect the inputs and output capabilities from the module caps. static gboolean gst_ml_video_super_resolution_set_caps (GstBaseTransform * base, GstCaps * incaps, GstCaps * outcaps) Copy to clipboard 2. Fix negotiated caps. static GstCaps * gst_ml_video_super_resolution_fixate_caps (GstBaseTransform * base, GstPadDirection direction, GstCaps * incaps, GstCaps * outcaps) Copy to clipboard 3. Check whether specified caps are valid or not like video/x-raw, video/x-text, format, height and width. static gboolean caps_has_feature (const GstCaps * caps, const gchar * feature) Copy to clipboard 4. Convert static caps to regular caps. static GstCaps * gst_ml_video_super_resolution_src_caps (void) Copy to clipboard 5. Convert static caps to regular caps. static GstCaps * gst_ml_video_super_resolution_sink_caps (void) Copy to clipboard - Transform 1. Transform sink caps to source caps and vice versa. static GstCaps * gst_ml_video_super_resolution_transform_caps (GstBaseTransform * base, GstPadDirection direction, GstCaps * caps, GstCaps * filter) Copy to clipboard 2. After inference, the postprocessing function (`gst_ml_video_superresolution_module_execute`) will be called based on the module. gst_ml_video_super_resolution_transform (GstBaseTransform * base, GstBuffer * inbuffer, GstBuffer * outbuffer) Copy to clipboard - Property 1. Set plugin properties, like module and constant. static void gst_ml_video_super_resolution_set_property (GObject * object, guint prop_id, const GValue * value, GParamSpec * pspec) Copy to clipboard 2. Read plugin properties, like module and constant. static void gst_ml_video_super_resolution_get_property (GObject * object, guint prop_id, GValue * value, GParamSpec * pspec) Copy to clipboard - De-init 1. Free all allocated buffers. static void gst_ml_video_super_resolution_finalize (GObject * object) Copy to clipboard - Display text/bounding boxes (based on use case only, not required for super resolution) 1. Overlay result on output frame static gboolean gst_ml_video__fill_video_output(GstMLVideoDetection * detection, GArray * predictions, GstBuffer * buffer); Copy to clipboard 4. Update the plugin name in the `gst-plugins-mlvsuperresolution/CMakeLists.txt` file. Use the existing plugin as a reference. [gst-plugin-mlvsegmentation/CMakeLists.txt imsdk.lnx.2.0.0.r2-rel · CodeLinaro / le / platform / vendor / qcom-opensource / gst-plugins-qti-oss · GitLab](https://git.codelinaro.org/clo/le/platform/vendor/qcom-opensource/gst-plugins-qti-oss/-/blob/imsdk.lnx.2.0.0.r2-rel/gst-plugin-mlvsegmentation/CMakeLists.txt?ref_type=heads) Reuse the `config.h.in` file from the existing plugin: [gst-plugin-mlvsegmentation/config.h.in imsdk.lnx.2.0.0.r2-rel · CodeLinaro / le / platform / vendor / qcom-opensource / gst-plugins-qti-oss · GitLab](https://git.codelinaro.org/clo/le/platform/vendor/qcom-opensource/gst-plugins-qti-oss/-/blob/imsdk.lnx.2.0.0.r2-rel/gst-plugin-mlvsegmentation/config.h.in?ref_type=heads) ## Overview of Module - Each Model contains specific postprocessing steps, which need to be performed on inferenced buffers and update output buffers accordingly like populating RGB pixels for super resolution and segmentation or calculating bounding boxes for detection. - These post processing steps are defined inside “<module-name>.c”. Module directory contains list of files like below structure > > > □ modules > > > “gst-plugins-mlvsuperresolution/modules/ml-vsuperresolution-<module- > name1>.c” > > > “gst-plugins-mlvsuperresolution/modules/ml-vsuperresolution-<module- > name2>.c” > > > Here <module-name> represents the model’s name like srnet for > superresolution. ## Steps to create the srnet module 1. Create the module header `gst-plugins-qti-oss/gst-plugin-base/gst/ml/ml-module-video-super-resolution.h`. Use the existing plugin as a reference: [gst-plugin-base/gst/ml/ml-module-video-super-resolution.h imsdk.lnx.2.0.0.r2-rel · CodeLinaro / le / platform / vendor / qcom-opensource / gst-plugins-qti-oss · GitLab](https://git.codelinaro.org/clo/le/platform/vendor/qcom-opensource/gst-plugins-qti-oss/-/blob/imsdk.lnx.2.0.0.r2-rel/gst-plugin-base/gst/ml/ml-module-video-super-resolution.h?ref_type=heads) 1. Import the required `gst` headers. 2. Create the `execute` function. GST_API gboolean gst_ml_module_video_super_resolution_execute(GstMLModule * module, GstMLFrame * mlframe, GstVideoFrame * vframe); Copy to clipboard This function is required to call the postprocessing function (`gst_ml_module_process`) which is defined in the `ml-vsuperresolution-.c` file. 2. Create module the `gst-plugins-mlvsuperresolution/modules/ml-vsuperresolution-srnet.c` file. Use the existing pluging as a reference: [gst-plugin-mlvsegmentation/modules/ml-vsegmentation-deeplab-argmax.c · imsdk.lnx.2.0.0.r1-rel · CodeLinaro / le / platform / vendor / qcom-opensource / gst-plugins-qti-oss · GitLab](https://git.codelinaro.org/clo/le/platform/vendor/qcom-opensource/gst-plugins-qti-oss/-/blob/imsdk.lnx.2.0.0.r1-rel/gst-plugin-mlvsegmentation/modules/ml-vsegmentation-deeplab-argmax.c?ref_type=heads) 1. Import the `modules/ml-video-superresolution-module.h` header. 2. Define the model output caps. #define GST_ML_MODULE_CAPS \ "neural-network/tensors, " \ "type = (string) { INT8, UINT8, INT32, FLOAT32 }, " \ "dimensions = (int) < <1, [32, 4096], [32, 4096]> >; " \ "neural-network/tensors, " \ "type = (string) { INT8, UINT8, INT32, FLOAT32 }, " \ "dimensions = (int) < <1, [32, 4096], [32, 4096], [1, 3]> >" Copy to clipboard **For Example** > > > 1. Download [QuickSRNetLarge-Quantized- Qualcomm AI Hub](https://aihub.qualcomm.com/iot/models/quicksrnetlarge_quantized) model from AI Hub > 2. Check the output dimensions of the TFLite model (See [Integrate AI Hub models in an application](https://docs.qualcomm.com/doc/80-70017-15B/topic/integrate-ai-hub-models.html)). > > - Format: `dtype[N x H X W X C]` > - In the above module output caps `<1, [32, 4096], [32, 4096], [1, 3]>` > > - `N = 1`: Model output batch size > - `H = [32,4096]`: Model output height should be in this range > - `W= [32,4096]`: Model output width should be in this range > - `C = [1,3]`: Model output channels should be in this range > - `dtype`: `int8` for quantized models and `float32` for non-quantized models > 3. For quantized models, check the output node for scale and offset. > > > Format: `scale*(q-offset)` 3. Implement following functions - Init gpointer gst_ml_module_open (void); Copy to clipboard - De-init void gst_ml_module_close (gpointer instance); Copy to clipboard - Caps GstCaps *gst_ml_module_caps (void); Copy to clipboard - Configure gboolean gst_ml_module_configure (gpointer instance, GstStructure * settings); Copy to clipboard Check for module dytpe, scale, and offset. - De-quantize static inline gdouble gst_ml_module_get_dequant_value (void *data, GstMLType mltype, guint idx, gdouble offset, gdouble scale); Copy to clipboard Scale and offset values need to passed for quantized models - Postprocess gboolean gst_ml_module_process (gpointer instance, GstMLFrame *mlframe, gpointer output); Copy to clipboard - Inferenced buffer indata = GST_ML_FRAME_BLOCK_DATA (mlframe, 0); Copy to clipboard - Output buffer to be filled with postprocessed data outdata = GST_VIDEO_FRAME_PLANE_DATA (vframe, 0); Copy to clipboard - Data type of the model mltype = GST_ML_FRAME_TYPE (mlframe); Copy to clipboard - Postprocess (for super resolution) for (row = 0; row < GST_VIDEO_FRAME_HEIGHT (vframe row++) { for (column = 0; column < GST_VIDEO_FRAME_WIDTH (vframe); column++) { // Calculate the destination index. idx = (((row * GST_VIDEO_FRAME_WIDTH (vframe)) + column) * bpp) + (row * padding) outdata[idx] = gst_ml_module_get_dequant_value(indata, mltype, idx, submodule->qoffsets[0], submodule->qscales[0]) * 255; outdata[idx + 1] = gst_ml_module_get_dequant_value(indata, mltype, idx + 1, submodule->qoffsets[0], submodule->qscales[0]) * 255; outdata[idx + 2] = gst_ml_module_get_dequant_value(indata, mltype, idx + 2, submodule->qoffsets[0], submodule->qscales[0]) * 255; if (bpp == 4) outdata[idx + 3] = 0; } } Copy to clipboard 4. Update the module name in the `gst-plugins-mlvsuperresolution/modules/CMakeLists.txt` file. Use the existing plugin as a reference: [gst-plugin-mlvsegmentation/modules/CMakeLists.txt · imsdk.lnx.2.0.0.r1-rel · CodeLinaro / le / platform / vendor / qcom-opensource / gst-plugins-qti-oss · GitLab](https://git.codelinaro.org/clo/le/platform/vendor/qcom-opensource/gst-plugins-qti-oss/-/blob/imsdk.lnx.2.0.0.r1-rel/gst-plugin-mlvsegmentation/modules/CMakeLists.txt?ref_type=heads) ## Steps to update BitBake files for plugin compilation 1. Follow [Create your Qualcomm IM SDK plugin](https://docs.qualcomm.com/doc/80-70017-15B/topic/create-your-imsdk-plugin.html) for instructions on compiling and installing the superresolution plugin. 2. Setup the Wayland display. ssh root@[IP address of the target device] Copy to clipboard Note If prompted, enter `oelinux123` as the password for the ssh shell. export XDG_RUNTIME_DIR=/dev/socket/weston && export WAYLAND_DISPLAY=wayland-1 Copy to clipboard Reference GStreamer pipeline to use qtimlvsuperresolution plugin gst-launch-1.0 -e --gst-debug=2,qtimltflite:7 filesrc location=/opt/video.mp4 ! qtdemux ! queue ! h264parse ! \ v4l2h264dec capture-io-mode=5 output-io-mode=5 ! queue ! tee name=split \ split. ! queue ! qtivcomposer name=mixer \ sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" \ sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" \ ! queue ! waylandsink sync=false fullscreen=true \ split. ! qtimlvconverter ! queue ! \ qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so \ external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=/opt/quicksrnetlarge_quantized.tflite ! queue ! \ qtimlvsuperresolution module=srnet constants="qsrnetlarge,q-offsets=<-128.0>,q-scales=<0.0039139483124017715>;" ! \ video/x-raw,width=512,height=512,format=RGB ! \ queue ! mixer. Copy to clipboard Note Input video resolution must be 128x128x3. As the `quicksrnetlarge_quantized.tflite` model input dimensions are 128x128x3. Last Published: Jan 21, 2026 [Previous Topic General guidelines for developing a GStreamer plugin](https://docs.qualcomm.com/bundle/publicresource/80-70017-15B/topics/general-guidelines-for-developing-gstreamer-plugin.md) [Next Topic AI Hub: Qualcomm IM SDK](https://docs.qualcomm.com/bundle/publicresource/80-70017-15B/topics/ai-hub-qualcomm-im-sdk.md)