# Run LiteRT sample applications
The LiteRT framework provides sample applications that you can use to do the following:
- Run an arbitrary LiteRT model
- Perform benchmarking
## Download models and sample images
The sample applications in [Get started](https://docs.qualcomm.com/doc/80-70018-54/topic/getting-started.html#getting-started) use the label\_image sample application provided by the LiteRT framework, which can run any classification models. For example, MobileNet v1, v2.
Before you begin, ensure that you have the following:
- Ubuntu 22.04 host computer
- Qualcomm Linux development kit
To use the sample applications, download the following:
- Sample model
- Corresponding file containing labels
- Sample image
The sample applications use the MobileNet v1 model, which is trained on an ImageNet data set with 1000 classes as an example. MobileNet v1 demonstrates a model trained to classify an image.
For instructions on how to download and copy the models, label files, and the sample image to the device, see [Get started](https://docs.qualcomm.com/doc/80-70018-54/topic/getting-started.html#getting-started).
## Run a LiteRT model using an available delegate
The LiteRT open-source framework provides the label\_image sample application to
run a LiteRT model using an available delegate. The source code for the label\_image
sample application is available on the [TensorFlow GitHub repository](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/label_image).
The label\_image sample application is cross-compiled along with the LiteRT library and installed on the target device.
The following example demonstrates how to run LiteRT models using the available LiteRT delegates:
Note
You can also use a delegate with the label\_image sample application.
- To use the XNNPACK delegate, run the following commands:
ssh root@[ip-addr]
cd /etc/artifacts
Copy to clipboard
label_image -l /etc/artifacts/labels.txt -i /etc/artifacts/grace_hopper.bmp -m /etc/artifacts/mobilenet_v1_1.0_224_quant.tflite -c 10 -p 1 --xnnpack_delegate 1
Copy to clipboard
- To use the GPU delegate, run the following commands:
ssh root@[ip-addr]
cd /etc/artifacts
Copy to clipboard
label_image -l /etc/artifacts/labels.txt -i /etc/artifacts/grace_hopper.bmp -m /etc/artifacts/mobilenet_v1_1.0_224_quant.tflite -c 10 -p 1 --gl_backend 1
Copy to clipboard

**Figure: Performance statistics for GPU delegate creation**
### Benchmark LiteRT model performance
The open-source LiteRT provides a tool to benchmark model execution on hardware using delegates. This tool is available along with other artifacts installed on the device.
This benchmarking tool measures and calculates statistics for the following performance metrics:
- Initialization time
- Inference time of the Warm-up state
- Inference time of the Steady state
- Memory usage during initialization
- Overall memory usage
Before you begin, ensure that the downloaded models are in the `/etc/artifacts/` directory on the target device.
To perform benchmarking, do the following:
- To benchmark models using the XNNPACK delegate, run the following commands:
ssh root@[ip-addr]
cd /etc/artifacts
Copy to clipboard
benchmark_model --graph=/etc/artifacts/mobilenet_v1_1.0_224_quant.tflite --enable_op_profiling=true --use_xnnpack=true --num_threads=4 --max_secs=300 --profiling_output_csv_file=/etc/artifacts/mobilenet_v1_1.0_224_quant_xnnpack_performance.csv
Copy to clipboard
- To benchmark models using the GPU delegate, run the following commands:
ssh root@[ip-addr]
cd /etc/artifacts/
Copy to clipboard
benchmark_model --graph=/etc/artifacts/mobilenet_v1_1.0_224_quant.tflite --enable_op_profiling=true --use_gpu=true --num_runs=100 --warmup_runs=10 --max_secs=300 --profiling_output_csv_file==/etc/artifacts/mobilenet_v1_1.0_224_GPU_Delegate_performance.csv
Copy to clipboard

**Figure: Benchmark model tool statistics for GPU**
## Run a QNN delegate using an external delegate
The QNN delegate relies on the Qualcomm AI Engine direct API and its back ends to
speed up models on the Adreno GPU and the Hexagon Tensor Processor.
To run the QNN delegate using the external delegate interface, ensure that the following libraries are available on the device:
- `libQnnTFLiteDelegate.so` QNN delegate library
- Libraries from the Qualcomm AI Engine direct SDK
As part of the external delegate interface, `libQnnTFLiteDelegate.so` is available as an external delegate library to tools. After loading the delegate library, you can customize the model execution to use a specific back end through external delegate options.
For example:
- Use the `libQnnGpu.so` back-end library to run the QNN delegate on the GPU.
- Use the `libQnnHtp.so` back-end library to run models using the QNN delegate on the Hexagon Tensor Processor.
To benchmark a model on the Hexagon Tensor Processor, run the model through the QNN external delegate interface.
Use the following command to run inference:
benchmark_model --graph=/etc/artifacts/mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=/usr/lib/libQnnTFLiteDelegate.so --external_delegate_options='backend_type:htp;library_path:/usr/lib/ libQnnHtp.so;skel_library_dir:/usr/lib/rfsa/ adsp;htp_precision:0;htp_performance_mode:2'
Copy to clipboard
For more details, see [External delegate options for QNN delegate](https://docs.qualcomm.com/doc/80-70018-54/topic/sample-applications.html#external-delegate-options-for-qnn-delegate).
The figure highlights the following statistics presented by the benchmark\_model tool.
- Successful creation of the delegate or not
- Average inference time that the model took to run on the hardware using a delegate
- Memory footprint of the model execution

**Figure: Tool statistics: benchmark\_model**
**Known issue**
The LiteRT native sample application (label\_image) might crash during inferencing on the GPU or external delegate.
### External delegate options for QNN delegate
The external delegate interface dynamically loads the Qualcomm AI Engine direct
delegate. Therefore, it doesn’t have static information about the delegate options.
The external delegate sends strings as key and value pairs to the Qualcomm AI Engine direct delegate, which parses them as options. Therefore, the application using the external delegate interface must determine the accepted key and value option strings beforehand.
The following table lists the key value option strings that are available in the Qualcomm AI Engine direct delegate.
Table: Key value option strings
| Option key | Option value | Default value | Mandatory | Description |
| --- | --- | --- | --- | --- |
| backend\_type | GPU and Hexagon Tensor Processor | NA | Yes | The back-end Qualcomm AI Engine direct library used for opening and running the graph. |
| gpu\_precision | 0, 1, 2, 3 | 2 = Float16 for best performance | No | Precision for the GPU back end that defines the optimization levels of the graph tensors that are either input or output tensors.
>
>
>
0: Obey precisions specified in the LiteRT graph
1: Float32
2: Float16
3: Hybrid
0: Default
1: High
2: Normal
3: Low
0: Default
1: Sustained high performance
2: Burst
3: High performance
4: Power saver
5: Low-power saver
6: High-power saver
7: Low balance
8: Balance
0: Quantized precision
0: Optimize for inference
1: Optimize to prepare the model
0: Manual
1: Auto
0: Don’t enable HMX for short-depth conv2d.
1: Enable HMX for short-depth conv2d.
0: Don’t enable integration of ReLU into conv2d.
1: Enable integration of ReLU into conv2d.