# Architecture
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html)
The TensorFlow Lite framework runs models on devices with low-power requirements,
such as mobile, embedded, and edge platforms by optimizing them for latency, model size, and
power consumption.
The framework runs models with the help of delegates. Delegates are software layers that
use libraries written to execute a neural network model efficiently on a specific
hardware.
Figure : TensorFlow Lite Runtime architecture
## TensorFlow Lite Runtime
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html)
The TensorFlow Lite on-device inference loads the model into an interpreter, which
parses the model and uses a delegate to run it.
The TensorFlow Lite on-device inference does the following:
1. The inference loads the TensorFlow Lite model into a TensorFlow Lite interpreter
interface, which parses the model to identify neural network operators present
within the model.
2. The interpreter interface is further configured to run the model by using a
delegate.
3. The interpreter invokes a model inference on the provided inputs and saves the
corresponding outputs of model inference into the buffers provided to the
interpreter interface.
Qualcomm supports executing TensorFlow Lite models on the following accelerators using
delegates:
- CPU
- Adreno GPU
- Hexagon Tensor Processor
The following table lists the delegates and its accelerators:
Table : Supported delegates and accelerators
| Delegate | Acceleration |
| --- | --- |
| XNNPACK delegate | CPU |
| GPU delegate | GPU |
| Qualcomm^®^ AI Engine direct delegate (Qualcomm^®^ Neural Network (QNN) delegate) | CPU, GPU, and Hexagon Tensor Processor |
## Delegates for TensorFlow Lite
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html)
Delegates help you offload the TensorFlow Lite graph execution to the CPU, GPU, and
the Hexagon Tensor Processor hardware accelerators.
Currently, the following delegates are supported.
### XNNPACK delegate for CPU
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html)
The XNNPACK delegate uses the XNNPACK library to accelerate TensorFlow Lite models on
CPUs efficiently.
XNNPACK is an open-source library from Google, which does the following:
- Provides an optimized implementation of neural network operators to run on Arm
CPUs
- Uses low-level CPU instructions such as the Arm^®^ Neon™ instruction set to
optimize operators for efficient execution
The XNNPACK delegate can run models in both 32‑bit floating-point and INT8 formats. For
more information, see [XNNPACK back-end for TensorFlow Lite](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/xnnpack/README.md).
### GPU delegate
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html)
The GPU open-source delegate provides acceleration on various vendor-specific GPUs,
including the Adreno GPU.
TensorFlow Lite can use the GPU delegate to improve the parallel-processing power of
GPUs, which makes inferencing faster. The GPU delegate uses OpenCL kernels to run neural
network ops within a TensorFlow Lite model execution graph on the GPU.
The GPU delegate is cross-compiled by default along with the TensorFlow Lite library and
is optimized to run the following TensorFlow Lite models on the Adreno GPU:
- 16‑bit floating-point
- 32‑bit floating-point
For more information, see [GPU delegates for TensorFlow Lite](https://www.tensorflow.org/lite/performance/gpu).
### QNN delegate
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70015-54/topic/arch.html)
The QNN delegate is a proprietary delegate for vendor-specific hardware acceleration
to accelerate TensorFlow Lite models. The QNN delegate is designed based on the [external delegate interface](https://ai.google.dev/edge/litert/performance/implementing_delegate#option_2_leverage_external_delegate) of TensorFlow Lite.
You can use the QNN delegate to offload parts or the entire TensorFlow Lite model to
specialized Qualcomm hardware, such as the Adreno GPU and the Hexagon Tensor
Processor.
The QNN delegate improves the performance of model execution and power efficiency by
decreasing the CPU workload. The QNN delegate also uses the existing Qualcomm AI Engine
direct APIs and available back ends to accelerate models. For more information, see
[Qualcomm AI Engine direct](bundle/publicresource/topics/80-63442-50).
The QNN delegate can execute models in 32‑bit floating-point precision and INT8 precision
on the available hardware.
You can build applications using the following interfaces:
- Qualcomm AI Engine direct delegate interface
- TensorFlow Lite external delegate interface
You can access both the interfaces when using a standalone TensorFlow Lite application.
However, if you deploy your TensorFlow Lite models using the IM SDK, the qtimltflite
GStreamer plug-in for Qualcomm TensorFlow Lite Runtime uses the TensorFlow Lite external
delegate interface. For more information, see [Leverage external delegate](https://www.tensorflow.org/lite/performance/implementing_delegate#option_2_leverage_external_delegate).
The following figure shows the directory structure of QNN delegate libraries from the
Qualcomm AI Engine direct SDK:
Figure : QNN delegate directory structure
### Qualcomm AI Engine direct delegate interface
The Qualcomm AI Engine direct delegate interface, also known as the QNN delegate,
provides the `QnnTFLiteDelegate.h` header as an interface. You can
include this header as part of your application before linking it to the QNN
delegate library.
You can find a compatible QNN delegate library and QNN libraries placed in the
`aarch64-oe-linux-gcc11.2 cross-compiler` toolchain triplet
directory.
Table : QNN delegate acceleration support
| Back end name | Back end description | Target and library names | Library description |
| --- | --- | --- | --- |
| CPU | Back end for Arm CPU acceleration |
aarch64-oe-linux-gcc11.2
libQnnCpu.so
| `libQnnCpu.so`: CPU back-end library |
| GPU | Back end for the Adreno GPU hardware accelerator |
aarch64-oe-linux-gcc11.2
libQnnGpu.so
| `libQnnGpu.so`: GPU back-end library |
| Hexagon Tensor Processor | Back end for the Hexagon Tensor Processor hardware accelerator |
aarch64-oe-linux-gcc11.2
libQnnHtp.so
libQnnHtpPrepare.so
libQnnHtpV68Stub.so
hexagon-v68
libQnnHtpV68Skel.so
|
libQnnHtp.so: Library used for communication between the Arm CPU and the Hexagon Tensor Processor.
libQnnHtpPrepare.so: Hexagon Tensor Processor library running on the CPU side. This library is loaded to prepare the graph and optimize the execution graph at runtime.
libQnnHtpV68Stub.so: Hexagon Tensor Processor proxy library on the CPU side, communicating with the Skel library loaded on the Hexagon Tensor Processor.
libQnnHtpV68Skel.so: Hexagon Tensor Processor native library with optimized kernel/operator implementation for the Hexagon Tensor Processor.
|
Note:Select the appropriate libraries based on the Hexagon Tensor
Processor version of the Qualcomm Linux Development Kit. The versions are as
follows:
- QCS6490/QCS5430: Hexagon Tensor Processor v68
- QCS9075: Hexagon Tensor Processor v73
### TensorFlow Lite external delegate interface
In an external delegate interface, the application to run TensorFlow Lite models must
load the `libQnnTFLiteDelegate.so` QNN delegate library. The C/C++
application and the `libQnnTFLiteDelegate.so` delegate library have
no dependency on each other. Therefore, if the delegate changes, you do not have to
recompile the application.
In addition to the TensorFlow Lite Android C API, integrate the following into the
C/C++ Android application in the same way as the TensorFlow Lite Android C API:
- The external\_delegate.h header file
- The libexternal\_delegate.so shared library
The QNN delegate offers acceleration on the Hexagon Tensor Processor, GPU, and CPU.
To customize where and how to run models using the QNN delegate when using the
external delegate interface, you must provide additional external delegate options.
For instructions, see [External delegate options for QNN delegate](https://docs.qualcomm.com/doc/80-70015-54/topic/sample-applications.html#external-delegate-options-for-qnn-delegate).
Last Published: Oct 09, 2024
[Previous Topic
Get started](https://docs.qualcomm.com/bundle/publicresource/80-70015-54/topics/getting-started.md) [Next Topic
TensorFlow Lite developer workflow](https://docs.qualcomm.com/bundle/publicresource/80-70015-54/topics/tensorflow-lite-developer-workflow.md)