# LiteRT developer workflow
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
You can use an existing LiteRT model by downloading it from the open-source
community. Alternatively, you can convert a TensorFlow or Keras model to the LiteRT format
using specific tools. Once converted, you can run inference on a device and develop a custom
application for the LiteRT model.
Note: If you are using LiteRT models from Qualcomm AI Hub or other
sources, you may skip the tasks described in the LiteRT developer workflow.
Executing a LiteRT model on Qualcomm-specific hardware involves the following tasks.
## Convert a TensorFlow model to a LiteRT model
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
You can convert TensorFlow models to LiteRT models and optimize them for on-device
inference. For more details on LiteRT model conversion, see [Model conversion overview](https://ai.google.dev/edge/litert/models/convert).
LiteRT model conversion supports converting models to the following formats:
- 32‑bit floating-point precision
- 16‑bit floating-point precision
- UINT8/INT8 precision (quantizing models)
### Use an existing LiteRT model
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
You can deploy an existing LiteRT model from the open-source community using
LiteRT.
Qualcomm AI Hub publishes LiteRT models optimized for the Qualcomm Linux Development Kit.
For LiteRT models from Qualcomm, see [Qualcomm AI Hub](https://aihub.qualcomm.com/).
To download an optimized model from Qualcomm AI Hub, do the following:
1. Go to [AI Hub Model Zoo](https://aihub.qualcomm.com/iot/models).
2. In the left pane, filter the available models by selecting a chipset.
3. Select a model.
4. On the next page, select the TorchScript
>
TFLite path.
5. Click Download model.
Note: The downloaded model is pre-optimized and ready for
deployment.
Figure : Optimized LiteRT model on Qualcomm AI Hub
### Convert a TensorFlow or Keras model to the LiteRT format
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
The TensorFlow framework provides both Python APIs and a command-line interface (CLI)
tool to convert a TensorFlow or Keras model to the LiteRT format.
Table : TensorFlow model conversion methods
| Conversion method | Description |
| --- | --- |
| Python APIs | Converts, optimizes, and quantizes models to the LiteRT format |
| CLI tool | Converts models to the LiteRT format, but is suitable for basic model conversion only |
Note: The TensorFlow to LiteRT Python APIs offer more flexibility to
convert, optimize, and quantize models to suit your requirements.
### Convert and quantize using Python APIs
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
TensorFlow provides the following APIs to convert a TensorFlow SavedModel or a Keras
model to a LiteRT model.
Table : TensorFlow Python APIs to convert models
| API | Description |
| --- | --- |
| tf.lite.TFLiteConverter.from\_saved\_model() (recommended) | Converts a TensorFlow SavedModel |
| tf.lite.TFLiteConverter.from\_keras\_model() | Converts a Keras model |
### Convert a TensorFlow SavedModel (recommended)
The following example shows how to convert a TensorFlow model saved in the
saved\_model format to a LiteRT model:
import tensorflow as tf
# Convert the model
saved_model_dir = "/path/to/tf/model/in/saved_model/format"
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
# Save the model
with open("model.tflite", "wb") as f:
f.write(tflite_model)
Copy to clipboard
Note: The converted LiteRT model is not quantized and its data is
in 32‑bit floating-point precision.
### Convert a Keras model
The following example shows how to convert a Keras model to a LiteRT model:
import tensorflow as tf
# Create a model using high-level tf.keras.* APIs
model = tf.keras.models.Sequential([
tf.keras.layers.Dense(units=1, input_shape=[1]),
tf.keras.layers.Dense(units=16, activation='relu'),
tf.keras.layers.Dense(units=1)
])
# compile the model
model.compile(optimizer='sgd', loss='mean_squared_error')
# train the model
model.fit(x=[-1, 0, 1], y=[-3, -1, 1], epochs=5)
# Convert the model to LiteRT
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
# Save the model
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
Copy to clipboard
Note: The converted LiteRT model is not quantized and its data is
in 32‑bit floating-point precision.
### Quantize models
After converting a model to the LiteRT format, you can quantize it. Quantization in
neural network models involves the following steps:
1. Quantize weights and biases: These are already part of the trained model and can
be quantized without additional information. Hence, quantizing weights and
biases is a static step.
2. Quantize activation layers: The ranges for the activation layer output depend on
the input image during forward propagation. Therefore, a set of sample inputs,
known as calibration or representative data sets, is required to quantize these
layers and identify the minimum/maximum ranges
To quantize a TensorFlow floating-point model to a quantized LiteRT model, LiteRT
provides posttraining quantization techniques. For more information, see [Posttraining quantization](https://ai.google.dev/edge/litert/models/post_training_quantization).
### Posttraining quantization
LiteRT supports two types of posttraining quantizations:
- Posttraining dynamic range quantization
- Posttraining full-integer quantization
### Posttraining dynamic range quantization
In posttraining dynamic range quantization, weights and biases are statically
quantized from floating-point precision to fixed-point integer 8‑bit precision. The
activation layer ranges remain in 32‑bit floating-point precision.
To reduce latencies during inference, dynamic-range operators do the following:
- Quantize activations based on their ranges to fixed-point integer 8‑bit
precision
- Perform computations with 8‑bit weights and activations
Note: No additional calibration data are needed in this step
because only weights are quantized.
The following script converts and quantizes a TensorFlow model to a LiteRT model:
import tensorflow as tf
from tensorflow import keras
converter = tf.lite.TFLiteConverter.from_saved_model(exp_model_path)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
save_name = 'quantized_model.tflite'
print('Saving Dynamic Quantized LiteRT model ..................')
with open(save_name, 'wb') as f:
f.write(tflite_model)
Copy to clipboard
### Posttraining full-integer quantization
In full-integer quantization, a representative data set is used to perform
quantization for the activation layers within the model.
The following script converts and quantizes a TensorFlow model to a LiteRT model. It
generates a full-integer quantized model that is more suitable for fixed-point
integer hardware, such as the Hexagon Tensor Processor on the Qualcomm Linux
Development Kit.
import tensorflow as tf
def representative_dataset():
for data in dataset:
yield {
"image": data.image,
"bias": data.bias,
}
saved_model_dir = "/path/to/saved/model"
# prepare converter by loading model in saved_model format.
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# Set representative dataset used for quantization.
converter.representative_dataset = representative_dataset
# For full-integer quantization, set target_spec supported_ops to TFLITE_BUILTINS_INT8.
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8 # or tf.uint8
converter.inference_output_type = tf.int8 # or tf.uint8
# Convert model
tflite_quant_model = converter.convert()
save_name = 'quantized_model_int8.tflite'
print('Saving Quantized LiteRT model ..................')
with open(save_name, 'wb') as f:
f.write(tflite_model)
Copy to clipboard
Note: `supported_ops` in converter
`target_spec` is set to
`tf.lite.OpsSet.TFLITE_BUILTINS_INT8`.
### Convert using offline converter tool (CLI)
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
The tflite\_convert TensorFlow Lite converter tool is included with the TensorFlow pip
package and can be used offline for TensorFlow versions 2.x and later.
The tflite\_convert tool accepts the following input in the CLI:
tflite_convert --help
optional arguments:
-h, --help show this help message and exit
--output_file OUTPUT_FILE
Full filepath of the output file.
--saved_model_dir SAVED_MODEL_DIR
Full path of the directory containing the SavedModel.
--keras_model_file KERAS_MODEL_FILE
Full filepath of HDF5 file containing tf.Keras model.
--saved_model_tag_set SAVED_MODEL_TAG_SET
Comma-separated set of tags identifying the MetaGraphDef within the SavedModel to analyze. All tags must be present. To pass in an empty
tag set, pass in "". (default "serve")
--saved_model_signature_key SAVED_MODEL_SIGNATURE_KEY
Key identifying the SignatureDef containing inputs and outputs. (default DEFAULT_SERVING_SIGNATURE_DEF_KEY)
--enable_v1_converter
Enables the TensorFlow V1 converter in 2.0
Copy to clipboard
### Convert a SavedModel
To convert a typical TensorFlow model in the saved\_model format using the
tflite\_convert tool, run the following command:
tflite_convert \ --saved_model_dir=/tmp/mobilenet_saved_model \ --output_file=/tmp/mobilenet.tflite \ --saved_model_tag_set=serve \
--saved_model_signature_key="serving_default"
Copy to clipboard
### Convert a Keras H5 model
To convert a Keras model using the tflite\_convert tool, run the following
command:
tflite_convert \
--keras_model_file=/tmp/mobilenet_keras_model.h5 \
--output_file=/tmp/mobilenet.tflite
Copy to clipboard
Note: The tflite\_convert tool is suitable for basic purposes only.
For posttraining integer quantization, it is recommended to use Python APIs.
## Create an application and run inference
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
You can use the LiteRT C++ APIs to create an application, load a LiteRT model, and
execute it on hardware using delegates.
Typically, an application created using C++ APIs to run a LiteRT model involves the
following steps:
Figure : Workflow to create an application and run a LiteRT model
### Load a LiteRT model
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
A LiteRT model is a FlatBuffers file that contains information on model operators and
any associated weights and biases.
The contents of the FlatBuffers file include the following:
- Tensors (input and outputs of each op)
- Buffers (weights and biases)
- Ops that create an execution graph
The LiteRT framework provides APIs to do the following:
- Load a LiteRT model file
- Unpack all the content of the FlatBuffers file into memory
You can use the following APIs to load a LiteRT model for inference:
include
include
include "tensorflow/lite/interpreter.h"
include "tensorflow/lite/kernels/register.h"
include "tensorflow/lite/model.h"
include "tensorflow/lite/optional_debug_tools.h"
std::unique_ptr model;
model = tflite::FlatBufferModel::BuildFromFile(model_name.c_str());
if (!model) {
std::cerr << "Failed to mmap model " << model_name << std::endl;
exit(-1);
}
Copy to clipboard
### Create a LiteRT interpreter
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
Using the TensorFlow C/C++ APIs, you can build an interpreter to run the
model.
The interpreter interface helps you to do the following:
- Configure model execution on a chosen delegate.
- Allocate the memory needed to for forward propagation.
The following example code demonstrates how you can create an interpreter. You can
configure the interpreter instance to use a specific delegate and perform forward
propagation.
//Build the interpreter with the InterpreterBuilder.
//Note: all Interpreters should be built with the InterpreterBuilder,
// which allocates memory for the Interpreter and does various set up
// tasks so that the Interpreter can read the provided model.
tflite::ops::builtin::BuiltinOpResolver resolver;
tflite::InterpreterBuilder builder(*model, resolver);
std::unique_ptr interpreter;
builder(&interpreter);
if (!interpreter) {
std::cerr << "Failed to construct interpreter on provided tflite model" << std::endl;
}
if (interpreter->AllocateTensors() != kTfLiteOk) {
std::cerr << "Failed to allocate tensors!" << std::endl;
exit(-1);
}
Copy to clipboard
### Prepare a model with a chosen delegate
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
After creating an interpreter and allocating the necessary memory to run the model,
prepare the model with a chosen delegate. This step creates an execution graph from the
previously loaded model and uses the underlying library to perform inference on the delegate
hardware.
The following example code creates the XNNPACK delegate for running a LiteRT model on the
Arm CPU. The delegate is created by calling the
`TfLiteXNNPackDelegateCreate(…)` API. You can also customize the
delegate using the Delegate Options API.
TfLiteDelegate *delegate = NULL;
TfLiteXNNPackDelegateOptions xnnpack_options =
TfLiteXNNPackDelegateOptionsDefault();
xnnpack_options.num_threads = num_threads;
TfLiteDelegate* xnnpack_delegate =
TfLiteXNNPackDelegateCreate(&xnnpack_options);
if (interpreter->ModifyGraphWithDelegate(xnnpack_delegate) != kTfLiteOk) {
// Report error and fall back to another delegate, or the default backend
}
Copy to clipboard
### Prepare input/output buffers
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
When building a standalone LiteRT application, it is essential to prepare input data,
such as camera frames, for the pipeline to execute LiteRT models.
Preprocessing operations, in the following cases for example, are important to ensure
that inference is done correctly:
- Resizing the input image to a resolution expected by the model
- Normalization
- Mean subtraction
### Run a model
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
To run inference on a model, you must invoke a delegate using the
`Invoke()` API. Before invoking this API, create the appropriate
input/output buffers and provide them to the interpreter.
After the inference is complete, you can parse the output from the output buffers of the
interpreter to generate the inference results.
An example of the `Invoke()` API executing a model using a delegate is as
follows:
// Run Inference
interpreter->Invoke()
Copy to clipboard
After the inference is completed, you can find the output tensors from the LiteRT
`Invoke()` API in the output buffers of the interpreter. To perform
further postprocessing on these outputs, you can parse them from the interpreter.
For a comprehensive example, see the label\_image example in the [TensorFlow GitHub repository](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/label_image).
For more information, see [LiteRT documentation](https://ai.google.dev/edge/litert).
## Develop a custom application
Source: [https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70017-54/topic/tensorflow-lite-developer-workflow.html)
To enhance the developer experience, the IM SDK provides the qtimltflite
GStreamer-based plug-in, which performs LiteRT model inference.
For more details, see the following:
- [IM SDK](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-50/overview.html) documentation
- [qtimltflite](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-50/qtimltflite.html) plug-in documentation
- [Develop your own application](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-15B/develop-your-own-application.html)
Last Published: Jan 06, 2025
[Previous Topic
LiteRT architecture](https://docs.qualcomm.com/bundle/publicresource/80-70017-54/topics/arch.md) [Next Topic
Sample applications](https://docs.qualcomm.com/bundle/publicresource/80-70017-54/topics/sample-applications.md)