# Run LiteRT sample applications
The LiteRT framework provides sample applications that you can use to do the following:
- Run an arbitrary LiteRT model
- Perform benchmarking
You can run LiteRT models using the label\_image sample applications through available delegates, such as the XNNPACK delegate or the GPU delegate, and perform benchmarking. To run models using the QNN delegate on the GPU or the Hexagon Tensor Processor, use the respective back-end libraries and external delegate options for the QNN delegate.
## Prerequisites to run LiteRT sample applications
The native LiteRT sample application uses the label\_image sample application provided by the LiteRT framework, which can run classification models, such as MobileNet v1 and v2.
Before you begin, ensure that you have the following:
- An Ubuntu 22.04 host computer
- A Qualcomm development kit
To use the sample applications, download the following:
- Sample model
- Corresponding file containing labels
- Sample image
For instructions on how to download and copy the sample model, label files, and the sample image to the device, see [Run a LiteRT model using a native LiteRT sample application](https://docs.qualcomm.com/doc/80-70022-54/topic/getting-started.html#run-a-tensorflow-lite-model-using-a-native-tensorflow-lite-sample-application).
The sample applications use the MobileNet v1 model, which is trained on an ImageNet data set with 1000 classes as an example. MobileNet v1 demonstrates a model trained to classify an image.
## Run a LiteRT model using an available delegate
The LiteRT open-source framework provides the label\_image sample application to
run a LiteRT model using an available delegate. The source code for the label\_image
sample application is available on the [TensorFlow GitHub repository](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/label_image).
The label\_image sample application is cross-compiled along with the LiteRT library and installed on the target device. The following examples demonstrate how to run LiteRT models using the available LiteRT delegates:
To use the XNNPACK delegate, run the following commands:
Tab Qualcomm Linux
Tab Ubuntu
ssh root@[ip-addr]
cd /etc/artifacts
Copy to clipboard
label_image -l /etc/artifacts/labels.txt -i /etc/artifacts/grace_hopper.bmp -m /etc/artifacts/mobilenet_v1_1.0_224_quant.tflite -c 10 -p 1 --xnnpack_delegate 1
Copy to clipboard
ssh ubuntu@[ip-addr]
Copy to clipboard
label_image -l /etc/artifacts/labels.txt -i /etc/artifacts/grace_hopper.bmp -m /etc/artifacts/mobilenet_v1_1.0_224_quant.tflite -c 10 -p 1 --xnnpack_delegate 1
Copy to clipboard
To use the GPU delegate, run the following commands:
Tab Qualcomm Linux
Tab Ubuntu
ssh root@[ip-addr]
cd /etc/artifacts
Copy to clipboard
label_image -l /etc/artifacts/labels.txt -i /etc/artifacts/grace_hopper.bmp -m /etc/artifacts/mobilenet_v1_1.0_224_quant.tflite -c 10 -p 1 --gl_backend 1
Copy to clipboard
ssh ubuntu@[ip-addr]
Copy to clipboard
label_image -l /etc/artifacts/labels.txt -i /etc/artifacts/grace_hopper.bmp -m /etc/artifacts/mobilenet_v1_1.0_224_quant.tflite -c 10 -p 1 --gl_backend 1
Copy to clipboard
The following figure shows the performance statistics for the GPU delegate created:
Tab Qualcomm Linux
Tab Ubuntu

**Figure: Performance statistics for GPU delegate creation**

**Figure: Performance statistics for GPU delegate creation**
For more information about how the XNNPACK and GPU delegates accelerate models, see the following:
- [Use the XNNPACK delegate to accelerate models on CPUs](https://docs.qualcomm.com/doc/80-70022-54/topic/arch.html#xnnpack-delegate)
- [Use the GPU delegate to accelerate models on GPUs](https://docs.qualcomm.com/doc/80-70022-54/topic/arch.html#gpu-delegate)
### Benchmark LiteRT model performance
The open-source LiteRT provides a tool to benchmark model execution on hardware using delegates. This tool is available along with other artifacts installed on the device.
This benchmarking tool measures and calculates statistics for the following performance metrics:
- Initialization time
- Inference time of the Warm-up state
- Inference time of the Steady state
- Memory usage during initialization
- Overall memory usage
Before you begin, ensure that the downloaded models are in the `/etc/artifacts/` directory on the target device.
To benchmark models using the XNNPACK delegate, run the following commands:
Tab Qualcomm Linux
Tab Ubuntu
ssh root@[ip-addr]
cd /etc/artifacts
Copy to clipboard
benchmark_model --graph=/etc/artifacts/mobilenet_v1_1.0_224_quant.tflite --enable_op_profiling=true --use_xnnpack=true --num_threads=4 --max_secs=300 --profiling_output_csv_file=/etc/artifacts/mobilenet_v1_1.0_224_quant_xnnpack_performance.csv
Copy to clipboard
ssh ubuntu@[ip-addr]
Copy to clipboard
benchmark_model --graph=/etc/artifacts/mobilenet_v1_1.0_224_quant.tflite --enable_op_profiling=true --use_xnnpack=true --num_threads=4 --max_secs=300 --profiling_output_csv_file=/etc/artifacts/mobilenet_v1_1.0_224_quant_xnnpack_performance.csv
Copy to clipboard
To benchmark models using the GPU delegate, run the following commands:
Tab Qualcomm Linux
Tab Ubuntu
ssh root@[ip-addr]
cd /etc/artifacts/
Copy to clipboard
benchmark_model --graph=/etc/artifacts/mobilenet_v1_1.0_224_quant.tflite --enable_op_profiling=true --use_gpu=true --num_runs=100 --warmup_runs=10 --max_secs=300 --profiling_output_csv_file==/etc/artifacts/mobilenet_v1_1.0_224_GPU_Delegate_performance.csv
Copy to clipboard
ssh ubuntu@[ip-addr]
Copy to clipboard
benchmark_model --graph=/etc/artifacts/mobilenet_v1_1.0_224_quant.tflite --enable_op_profiling=true --use_gpu=true --num_runs=100 --warmup_runs=10 --max_secs=300 --profiling_output_csv_file==/etc/artifacts/mobilenet_v1_1.0_224_GPU_Delegate_performance.csv
Copy to clipboard
The following figure shows the benchmark\_model tool execution statistics for the GPU delegate:
Tab Qualcomm Linux
Tab Ubuntu

**Figure: Benchmark model tool statistics for GPU**

**Figure: Benchmark model tool statistics for GPU**
## Run the QNN delegate using an external delegate
The QNN delegate relies on the Qualcomm AI Engine direct API and its back ends to speed up models on the Adreno GPU and the Hexagon Tensor Processor.
To run the QNN delegate using the external delegate interface, ensure that the following libraries are available on the device:
- `libQnnTFLiteDelegate.so` QNN delegate library
- Libraries from the Qualcomm AI Engine direct SDK
As part of the external delegate interface, `libQnnTFLiteDelegate.so` is available as an external delegate library to tools. After loading the delegate library, you can customize the model execution to use a specific back end through external delegate options.
For example:
- Use the `libQnnGpu.so` back-end library to run the QNN delegate on the GPU.
- Use the `libQnnHtp.so` back-end library to run models using the QNN delegate on the Hexagon Tensor Processor.
- Use the `libQnnDsp.so` back-end library to run models using the QNN delegate on the DSP.
To benchmark a model on the Hexagon Tensor Processor, run the model through the QNN external delegate interface. Use the following command to run inference:
Tab Qualcomm Linux
Tab Ubuntu
- QCS6490/QCS5430, IQ-9075, and QCS8275:
>
>
> benchmark_model --graph=/etc/artifacts/mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=/usr/lib/libQnnTFLiteDelegate.so --external_delegate_options='backend_type:htp;library_path:/usr/lib/libQnnHtp.so;skel_library_dir:/usr/lib/rfsa/adsp;htp_precision:0;htp_performance_mode:2'
> Copy to clipboard
- IQ-615:
>
>
> benchmark_model --graph=/usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=libQnnTFLiteDelegate.so --external_delegate_options='backend_type:dsp;library_path:/usr/lib/libQnnDsp.so;skel_library_dir:/usr/lib/dsp/adsp'
> Copy to clipboard
benchmark_model --graph=/etc/artifacts/mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=/usr/lib/libQnnTFLiteDelegate.so --external_delegate_options='backend_type:htp;library_path:/usr/lib/libQnnHtp.so;skel_library_dir:/usr/lib/rfsa/adsp;htp_precision:0;htp_performance_mode:2'
Copy to clipboard
For more information, see [External delegate options for QNN delegate](https://docs.qualcomm.com/doc/80-70022-54/topic/sample-applications.html#external-delegate-options-for-qnn-delegate).
The following figure highlights the following statistics presented by the benchmark\_model tool:
- Delegate creation status
- Average inference time that the model took to run on the hardware using a delegate
- Memory footprint of the model execution
Tab Qualcomm Linux
Tab Ubuntu
- *class* tabincludedirective
- 
**Figure: Tool statistics: benchmark\_model**
- *class* tabincludedirective
- 
**Figure: Tool statistics: benchmark\_model**
For more information about how the QNN delegates accelerates models, see [Use the QNN delegate to accelerate models on specific hardware](https://docs.qualcomm.com/doc/80-70022-54/topic/arch.html#qnn-delegate).
### External delegate options for the QNN delegate
The external delegate interface dynamically loads the Qualcomm AI Engine direct
delegate. Therefore, it doesn’t have static information about the delegate options.
The external delegate sends strings as key and value pairs to the Qualcomm AI Engine direct delegate, which parses them as options. Therefore, the application using the external delegate interface must determine the accepted key and value option strings beforehand.
The following table lists the key value option strings that are available in the Qualcomm AI Engine direct delegate:
Table: Key value option strings
| Option key | Option value | Default value | Mandatory | Description |
| --- | --- | --- | --- | --- |
| backend\_type | GPU and Hexagon Tensor Processor | NA | Yes | The back-end Qualcomm AI Engine direct library used for opening and running the graph. |
| gpu\_precision | 0, 1, 2, 3 | 2 = Float16 for best performance | No | Precision for the GPU back end that defines the optimization levels of the graph tensors that are either input or output tensors.
>
>
>
0: Obey precisions specified in the LiteRT graph
1: Float32
2: Float16
3: Hybrid
0: Default
1: High
2: Normal
3: Low
0: Default
1: Sustained high performance
2: Burst
3: High performance
4: Power saver
5: Low-power saver
6: High-power saver
7: Low balance
8: Balance
0: Quantized precision
0: Optimize for inference
1: Optimize to prepare the model
0: Manual
1: Auto
0: Don’t enable HMX for short-depth conv2d.
1: Enable HMX for short-depth conv2d.
0: Don’t enable integration of ReLU into conv2d.
1: Enable integration of ReLU into conv2d.