# LiteRT architecture
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html)
The LiteRT framework is designed to run models on devices with low-power
requirements, such as mobile, embedded, and edge platforms by optimizing them for latency,
model size, and power consumption.
The framework runs models with the help of delegates. Delegates are software layers that
use libraries to run a neural network model efficiently on specific hardware.
Figure : LiteRT architecture
## LiteRT on-device inference
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html)
The LiteRT on-device inference loads the model into an interpreter, which parses the
model and uses a delegate to run it.
The process includes the following:
1. The inference loads the LiteRT model into a LiteRT interpreter interface, which
parses the model to identify the neural network operators present in it.
2. The interpreter interface is further configured to run the model using a
delegate.
3. The interpreter invokes a model inference on the provided inputs and saves the
corresponding outputs into the buffers provided to the interpreter interface.
Qualcomm supports executing LiteRT models on the following accelerators using
delegates:
- CPU
- Adreno GPU
- Hexagon Tensor Processor
The following table lists the delegates and their accelerators.
Table : Supported delegates and accelerators
| Delegate | Acceleration |
| --- | --- |
| XNNPACK delegate | CPU |
| GPU delegate | GPU |
| Qualcomm^®^ AI Engine direct delegate (Qualcomm^®^ Neural Network (QNN) delegate) | CPU, GPU, and Hexagon Tensor Processor |
## Delegates for LiteRT
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html)
Delegates enable you to offload the LiteRT graph execution to hardware accelerators,
such as CPU, GPU, and the Hexagon Tensor Processor.
Currently, the following delegates are supported.
### XNNPACK delegate for CPU
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html)
The XNNPACK delegate uses the XNNPACK library to accelerate LiteRT models efficiently
on CPUs.
XNNPACK is an open-source library from Google, which does the following:
- Provides an optimized implementation of neural network operators for Arm CPUs
- Uses low-level CPU instructions, such as the Arm^®^ Neon™ instruction set,
to optimize operators for efficient execution
The XNNPACK delegate can run models in both 32‑bit floating-point and INT8 formats. For
more information, see [XNNPACK back-end for TensorFlow Lite](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/xnnpack/README.md).
### GPU delegate
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html)
The GPU open-source delegate accelerates LiteRT models on various vendor-specific
GPUs, including the Adreno GPU.
LiteRT can use the GPU delegate to improve the parallel-processing power of GPUs, which
makes inferencing faster. The GPU delegate uses OpenCL kernels to run neural network ops
within a LiteRT model execution graph on the GPU.
The GPU delegate is cross-compiled by default along with the LiteRT library and is
optimized to run the following LiteRT models on the Adreno GPU:
- 16‑bit floating-point
- 32‑bit floating-point
For more information, see [GPU delegates for LiteRT](https://www.tensorflow.org/lite/performance/gpu).
### QNN delegate
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html](https://docs.qualcomm.com/doc/80-70017-54/topic/arch.html)
The QNN delegate is a proprietary delegate designed for vendor-specific hardware
acceleration to accelerate LiteRT models. It is based on the [external delegate interface](https://ai.google.dev/edge/litert/performance/implementing_delegate#option_2_leverage_external_delegate) of LiteRT.
You can use the QNN delegate to offload parts or the entire LiteRT model to specialized
Qualcomm hardware, such as the Adreno GPU and the Hexagon Tensor Processor.
This delegate improves model execution performance and power efficiency by reducing the
CPU workload. It also uses the existing Qualcomm AI Engine direct APIs and available
back ends to accelerate models. For more information, see [Qualcomm
AI Engine direct SDK](bundle/publicresource/topics/80-63442-50).
The QNN delegate can execute models in both 32‑bit floating-point precision and INT8
precision on the available hardware.
You can build applications using the following interfaces:
- Qualcomm AI Engine direct delegate interface
- LiteRT external delegate interface
You can access both the interfaces when using a standalone LiteRT application. However,
if you deploy your LiteRT models using the IM SDK, the qtimltflite GStreamer plug-in for
Qualcomm TensorFlow Lite uses the QNN delegate. For more information, see [Leverage external delegate](https://ai.google.dev/edge/litert/performance/implementing_delegate#option_2_leverage_external_delegate).
The following figure shows the directory structure of QNN delegate libraries from the
Qualcomm AI Engine direct SDK:
Figure : QNN delegate directory structure
### Qualcomm AI Engine direct delegate interface
The Qualcomm AI Engine direct delegate interface, also known as the QNN delegate,
provides the `QnnTFLiteDelegate.h` header as an interface. You can
include this header in your application before linking it to the QNN delegate
library.
You can find the compatible QNN delegate library and QNN libraries in the
`aarch64-oe-linux-gcc11.2 cross-compiler` toolchain triplet
directory.
Table : QNN delegate acceleration support
| Back end name | Back end description | Target and library names | Library description |
| --- | --- | --- | --- |
| CPU | Back end for Arm CPU acceleration |
aarch64-oe-linux-gcc11.2
libQnnCpu.so
| `libQnnCpu.so`: CPU back-end library |
| GPU | Back end for the Adreno GPU hardware accelerator |
aarch64-oe-linux-gcc11.2
libQnnGpu.so
| `libQnnGpu.so`: GPU back-end library |
| Hexagon Tensor Processor | Back end for the Hexagon Tensor Processor hardware accelerator |
aarch64-oe-linux-gcc11.2
libQnnHtp.so
libQnnHtpPrepare.so
libQnnHtpV68Stub.so
hexagon-v68
libQnnHtpV68Skel.so
|
libQnnHtp.so: Library used for communication between the Arm CPU and the Hexagon Tensor Processor.
libQnnHtpPrepare.so: Hexagon Tensor Processor library running on the CPU side. This library is loaded to prepare the graph and optimize the execution graph at runtime.
libQnnHtpV68Stub.so: Hexagon Tensor Processor proxy library on the CPU side, communicating with the Skel library loaded on the Hexagon Tensor Processor.
libQnnHtpV68Skel.so: Hexagon Tensor Processor native library with optimized kernel/operator implementation for the Hexagon Tensor Processor.
|
Note:Select the appropriate libraries based on the Hexagon Tensor
Processor version of the Qualcomm Linux Development Kit. The versions are as
follows:
- QCS6490/QCS5430: Hexagon Tensor Processor v68
- QCS9075: Hexagon Tensor Processor v73
- QCS8275: Hexagon Tensor Processor v75
### LiteRT external delegate interface
To run LiteRT models using an external delegate interface, the application must load
the `libQnnTFLiteDelegate.so` QNN delegate library. The C/C++
application and the `libQnnTFLiteDelegate.so` delegate library have
no dependency on each other. Therefore, if the delegate changes, you do not have to
recompile the application.
In addition to the LiteRT Android C API, integrate the following into the C/C++
Android application in the same way as the LiteRT Android C API:
- The external\_delegate.h header file
- The libexternal\_delegate.so shared library
The QNN delegate provides acceleration on the Hexagon Tensor Processor, GPU, and CPU.
To customize where and how to run models using the QNN delegate with the external
delegate interface, you must provide additional external delegate options. For
instructions, see [External delegate options for QNN delegate](https://docs.qualcomm.com/doc/80-70017-54/topic/sample-applications.html#external-delegate-options-for-qnn-delegate).
Last Published: Jan 06, 2025
[Previous Topic
Get started](https://docs.qualcomm.com/bundle/publicresource/80-70017-54/topics/getting-started.md) [Next Topic
LiteRT developer workflow](https://docs.qualcomm.com/bundle/publicresource/80-70017-54/topics/tensorflow-lite-developer-workflow.md)