# Sample applications
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html](https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html)
The TensorFlow Lite framework provides sample applications that you can use to do the
following:
- Execute an arbitrary TensorFlow Lite model
- Perform benchmarking
## Download models and sample images
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html](https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html)
The sample applications in [Get started](https://docs.qualcomm.com/doc/80-70015-54/topic/getting-started.html) use the label\_image
sample application provided by the TensorFlow Lite framework, which can run any
classification models; for example, MobileNet v1, v2.
Before you begin, ensure that you have the following:
- Ubuntu 22.04 host machine
- Qualcomm Linux Development Kit
To use sample applications, download the following:
- Sample model
- Corresponding file containing labels
- Sample image
The sample applications use the MobileNet v1 model, which is trained on an ImageNet data
set with 1000 classes as an example. The MobileNet v1 model demonstrates a model trained
to classify an image.
For instructions on how to download and copy the models, label files, and the sample
image to the `/opt/` directory on the device, see [Get started](https://docs.qualcomm.com/doc/80-70015-54/topic/getting-started.html).
## Run a TensorFlow Lite model using an available delegate
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html](https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html)
The TensorFlow Lite open-source framework provides the label\_image sample application
to execute a TensorFlow Lite model using an available delegate. The source code for the
label\_image sample application is available on the [TensorFlow GitHub repository](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/label_image).
The label\_image sample application is cross-compiled along with the TensorFlow Lite
library and installed on the target device.
The following example demonstrates how to execute TensorFlow Lite models using the
available TensorFlow Lite delegates:
Note: You can also use a delegate with the label\_image sample
application.
- To use the XNNPACK delegate, run the following
commands:
ssh root@[ip-addr]
cd /opt
Copy to clipboard
label_image -l /opt/labels.txt -i /opt/grace_hopper.bmp -m /opt/mobilenet_v1_1.0_224_quant.tflite -c 10 -p 1 --xnnpack_delegate 1
Copy to clipboard
- To use the GPU delegate, run the following
commands:
ssh root@[ip-addr]
cd /opt
Copy to clipboard
label_image -l /opt/labels.txt -i /opt/grace_hopper.bmp -m /opt/mobilenet_v1_1.0_224_quant.tflite -c 10 -p 1 --gl_backend 1
Copy to clipboard
Figure : Performance statistics for GPU delegate creation
### Benchmark TensorFlow Lite model performance
The open-source TensorFlow Lite provides a tool to benchmark model execution on the
hardware using delegates. This tool is available along with other artifacts
installed on the device.
This benchmarking tool currently measures and calculates statistics for the following
performance metrics:
- Initialization time
- Inference time of the Warm-up state
- Inference time of the Steady state
- Memory usage during initialization time
- Overall memory usage
Before you begin, ensure that the downloaded models are in the `/opt/`
directory on the target device.
To perform benchmarking, do the following:
- To benchmark models using the XNNPACK delegate, run the following
commands:
ssh root@[ip-addr]
cd /opt/
Copy to clipboard
benchmark_model --graph=/opt/mobilenet_v1_1.0_224_quant.tflite -- enable_op_profiling=true --use_xnnpack=true --num_threads=4 --max_secs=300 --pofiling_output_csv_file=/opt/mobilenet_v1_1.0_224_quant_xnnpack_performance.csv
Copy to clipboard
- To benchmark models using the GPU delegate, run the following
commands:
ssh root@[ip-addr]
cd /opt/
Copy to clipboard
benchmark_model --graph=/opt/mobilenet_v1_1.0_224_quant.tflite --enable_op_profiling=true --use_gpu=true --num_runs=100 --warmup_runs=10 -- max_secs=300 --profiling_output_csv_file==/opt/mobilenet_v1_1.0_224_GPU_Delegate_performance.csv
Copy to clipboard
Figure : Benchmark model tool statistics for GPU
## Run a QNN delegate using an external delegate
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html](https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html)
The QNN delegate relies on the Qualcomm AI Engine direct API and its back ends to
speed up models on the Adreno GPU and the Hexagon Tensor Processor.
The following libraries must be available on the device to run the QNN delegate using the
external delegate interface:
- `libQnnTFLiteDelegate.so` QNN delegate library
- Libraries from the Qualcomm AI Engine direct SDK
As part of the external delegate interface, `libQnnTFLiteDelegate.so` is
provided as an external delegate library to tools. After the delegate library is loaded,
you can customize the execution to execute models using a specific back end through
external delegate options.
For example:
- The `libQnnGpu.so` back-end library is used for running the QNN
delegate on the GPU.
- The `libQnnHtp.so` back-end library is used for running models using
the QNN delegate on the Hexagon Tensor Processor.
To benchmark a model on the Hexagon Tensor Processor, run the model through the QNN
external delegate interface.
Use the following command to run inference:
benchmark_model --graph=/opt/mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=/usr/lib/libQnnTFLiteDelegate.so --external_delegate_options='backend_type:htp;library_path:/usr/lib/libQnnHtp.so;skel_library_dir:/usr/lib/rfsa/adsp;htp_precision:0;htp_performance_mode:2'Copy to clipboard
For more details, see [External delegate options for QNN delegate](https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html#external-delegate-options-for-qnn-delegate).
The figure highlights the following statistics presented by the benchmark\_model tool:
- Whether a delegate has been successfully created
- Average inference time the model took to run on the hardware using a delegate
- Memory footprint of the model execution
Figure : Tool statistics: benchmark\_model
### External delegate options for QNN delegate
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html](https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html)
The external delegate interface dynamically loads the Qualcomm AI Engine direct
delegate. Therefore, it does not have static information about the Qualcomm AI Engine direct
delegate options.
The external delegate sends strings as key and value pairs to the Qualcomm AI Engine
direct delegate, which parses them as options. The application using the external
delegate interface must determine the accepted key and value option strings
beforehand.
The following table lists the key value option strings that are available in the Qualcomm
AI Engine direct delegate:
Table : Key value option strings
| Option key | Option value | Default value | Mandatory | Description |
| --- | --- | --- | --- | --- |
| backend\_type | GPU and Hexagon Tensor Processor | NA | Yes | The back-end Qualcomm AI Engine direct library that is used for
opening and running the graph with. |
| gpu\_precision | 0, 1, 2, 3 | 2 = Float16 for best performance | No | Precision for the GPU back-end that defines the optimization levels
of the graph tensors that are either input or output tensors.
- 0: Obey precisions specified in the TensorFlow Lite graph
- 1: Float32
- 2: Float16
- 3: Hybrid
|
| gpu\_performance\_mode | 0, 1, 2, 3 | 0 = default | No | Flag to provide precision modes supported by the GPU back-end.
- 0: Default
- 1: High
- 2: Normal
- 3: Low
|
| htp\_performance\_mode | 0, 1, 2, 3, 4, 5, 6, 7, 8 | 0 = default | No | When performance\_mode is used, the delegate votes for the provided
performance level during inference and returns to a relaxed vote
after the inference has completed.
- 0: Default
- 1: Sustained high performance
- 2: Burst
- 3: High performance
- 4: Power saver
- 5: Low-power saver
- 6: High-power saver
- 7: Low balance
- 8: Balance
|
| htp\_pd\_session | Unsigned, signed | Unsigned | No | This flag defines the PD session of the Hexagon Tensor Processor
back-end. |
| htp\_precision | 0 | 0 = quantized precision | No | Flag to provide precision modes supported by the Hexagon Tensor
Processor back-end. The default precision mode supports quantized
networks. Other precision modes may only be supported on certain
SoCs.
|
| htp\_optimization\_strategy | 0, 1 | 0 = optimize for inference | No | Flag to select the optimization strategy used by the Hexagon Tensor
Processor back-end. The default optimization strategy optimizes the
graph for inference.
- 0: Optimize for inference
- 1: Optimize to prepare the model
|
| htp\_perf\_ctrl\_strategy | 0, 1 | 0 = manual | No | Flag to select the Hexagon Tensor Processor performance control
strategy. Manually vote the performance while the back end is
initialized and release the performance at the moment the back end
is destroyed.
|
| htp\_use\_conv\_hmx | 0, 1 | 1 = enable HMX for short depth conv2d | No | Flag to enable HMX for short depth conv2d. For more information, see
the C interface or qnn\_delegate.h.
- 0: Do not enable HMX for short-depth conv2d.
- 1: Enable HMX for short-depth conv2d.
|
| htp\_use\_fold\_relu | 0, 1 | 0 = do not fuse rectified linear unit (ReLU) into conv2d | No | Flag to enable fusion of ReLU into conv2d. For more information, see
the C interface or qnn\_delegate.h.
- 0: Do not enable fusion of ReLU into conv2d.
- 1: Enable fusion of ReLU into conv2d.
|
| library\_path | <Path to the back-end library file> | Default library associated with the chosen Qualcomm AI Engine direct
back-end | No | Optional parameter to override the Qualcomm AI Engine direct back-end
library. |
Last Published: Oct 09, 2024
[Previous Topic
TensorFlow Lite developer workflow](https://docs.qualcomm.com/bundle/publicresource/80-70015-54/topics/tensorflow-lite-developer-workflow.md) [Next Topic
Optional: Build TensorFlow Lite Runtime](https://docs.qualcomm.com/bundle/publicresource/80-70015-54/topics/build-qualcomm-linux-and-install-tensorflow-lite-runtime.md)