# LiteRT developer workflow
You can use an existing LiteRT model by downloading it from the open-source
community. Or you can convert a TensorFlow or Keras model to the LiteRT format
using specific tools. After converting the model, you can run inference on a device and develop a custom
application for the LiteRT model.
The following figure shows the tasks involved in using existing models, converting and quantizing models, and creating an application to run inference.
**Figure: LiteRT developer workflow**
Note
If you are using LiteRT models from Qualcomm AI Hub or other sources, you may skip the tasks described in the LiteRT developer workflow.
Running a LiteRT model on Qualcomm-specific hardware involves the following tasks.
## Convert a TensorFlow model to a LiteRT model
You can convert TensorFlow models to LiteRT models and optimize them for on-device inference. For more information about LiteRT model conversion, see [Model conversion overview](https://ai.google.dev/edge/litert/models/convert).
LiteRT model conversion supports converting models to the following formats:
- 32‑bit floating-point precision
- 16‑bit floating-point precision
- UINT8/INT8 precision (quantizing models)
### Use an existing LiteRT model
You can deploy an existing LiteRT model from the open-source community using
LiteRT.
Qualcomm AI Hub publishes LiteRT models optimized for the Qualcomm Linux development kit. For LiteRT models from Qualcomm, see [Qualcomm AI Hub](https://aihub.qualcomm.com/).
To download an optimized model from Qualcomm AI Hub, do the following:
1. Go to [Qualcomm AI Hub IOT models page](https://aihub.qualcomm.com/iot/models).
2. From the left pane, filter the available models by selecting a chipset.
3. Select a model.
4. On the next page, select the TorchScript > TFLite path.
5. Select Download model.
Note
The downloaded model is pre-optimized and ready for deployment.
**Figure: Optimized LiteRT model on Qualcomm AI Hub**
### Convert a TensorFlow or Keras model to the LiteRT format
The TensorFlow framework provides both Python APIs and a command-line interface (CLI)
tool to convert a TensorFlow or Keras model to the LiteRT format.
Table: TensorFlow model conversion methods
| Conversion method | Description |
| --- | --- |
| Python APIs | Converts, optimizes, and quantizes models to the LiteRT format |
| CLI tool | Converts models to the LiteRT format, but is suitable for basic model conversion only |
Note
The TensorFlow to LiteRT Python APIs offer more flexibility to convert, optimize, and quantize models to suit your requirements.
### Convert and quantize using Python APIs
TensorFlow provides the following APIs to convert a TensorFlow SavedModel or a Keras
model to a LiteRT model.
Table: TensorFlow Python APIs to convert models
| API | Description |
| --- | --- |
| tf.lite.TFLiteConverter.from\_saved\_model() (recommended) | Converts a TensorFlow SavedModel |
| tf.lite.TFLiteConverter.from\_keras\_model() | Converts a Keras model |
### Recommended: Convert a TensorFlow SavedModel
The following example shows how to convert a TensorFlow model saved in the saved\_model format to a LiteRT model:
import tensorflow as tf
# Convert the model
saved_model_dir = "/path/to/tf/model/in/saved_model/format"
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
# Save the model
with open("model.tflite", "wb") as f:
f.write(tflite_model)
Copy to clipboard
Note
The converted LiteRT model isn’t quantized, and its data is in 32‑bit floating-point precision.
### Convert a Keras model
The following example shows how to convert a Keras model to a LiteRT model:
import tensorflow as tf
# Create a model using high-level tf.keras.* APIs
model = tf.keras.models.Sequential([
tf.keras.layers.Dense(units=1, input_shape=[1]),
tf.keras.layers.Dense(units=16, activation='relu'),
tf.keras.layers.Dense(units=1)
])
# compile the model
model.compile(optimizer='sgd', loss='mean_squared_error')
# train the model
model.fit(x=[-1, 0, 1], y=[-3, -1, 1], epochs=5)
# Convert the model to LiteRT
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
# Save the model
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
Copy to clipboard
Note
The converted LiteRT model isn’t quantized, and its data is in 32‑bit floating-point precision.
## Quantize models
After converting a model to the LiteRT format, you can quantize it. Quantization in neural network models involves the following steps:
1. Quantize weights and biases: These are already part of the trained model and you can quantize them without additional information. Therefore, quantizing weights and biases is a static step.
2. Quantize activation layers: The ranges for the activation layer output depend on the input image during forward propagation. Therefore, a set of sample inputs, known as calibration or representative data sets, is necessary to quantize these layers and identify the minimum and maximum ranges.
To quantize a TensorFlow floating-point model to a quantized LiteRT model, LiteRT provides posttraining quantization techniques. For more information, see [Posttraining quantization](https://ai.google.dev/edge/litert/models/post_training_quantization).
### Posttraining quantization
LiteRT supports two types of posttraining quantizations:
- Posttraining dynamic range quantization
- Posttraining full-integer quantization
### Posttraining dynamic range quantization
In posttraining dynamic range quantization, weights and biases are statically quantized from floating-point precision to fixed-point integer 8‑bit precision. The activation layer ranges remain in 32‑bit floating-point precision.
To reduce latencies during inference, dynamic-range operators:
- Quantize activations based on their ranges to fixed-point integer 8‑bit precision
- Perform computations with 8‑bit weights and activations
Note
This step only quantizes weights and doesn’t need extra calibration data.
The following script converts and quantizes a TensorFlow model to a LiteRT model:
import tensorflow as tf
from tensorflow import keras
converter = tf.lite.TFLiteConverter.from_saved_model(exp_model_path)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
save_name = 'quantized_model.tflite'
print('Saving Dynamic Quantized LiteRT model ..................')
with open(save_name, 'wb') as f:
f.write(tflite_model)
Copy to clipboard
### Posttraining full-integer quantization
In full-integer quantization, a representative data quantizes the activation layers within the model.
The following script converts and quantizes a TensorFlow model to a LiteRT model. It generates a full-integer quantized model that’s more suitable for fixed-point integer hardware, such as the Hexagon Tensor Processor on the Qualcomm Linux development kit.
import tensorflow as tf
def representative_dataset():
for data in dataset:
yield {
"image": data.image,
"bias": data.bias,
}
saved_model_dir = "/path/to/saved/model"
# prepare converter by loading model in saved_model format.
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# Set representative dataset used for quantization.
converter.representative_dataset = representative_dataset
# For full-integer quantization, set target_spec supported_ops to TFLITE_BUILTINS_INT8.
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8 # or tf.uint8
converter.inference_output_type = tf.int8 # or tf.uint8
# Convert model
tflite_quant_model = converter.convert()
save_name = 'quantized_model_int8.tflite'
print('Saving Quantized LiteRT model ..................')
with open(save_name, 'wb') as f:
f.write(tflite_model)
Copy to clipboard
Note
`supported_ops` in converter sets `target_spec` to `tf.lite.OpsSet.TFLITE_BUILTINS_INT8`.
## Convert using offline converter tool
The TensorFlow pip package includes the tflite\_convert TensorFlow Lite offline converter tool (CLI), which you can use offline for TensorFlow versions 2.x and later.
The tflite\_convert tool accepts the following input in the CLI:
tflite_convert --help
optional arguments:
-h, --help show this help message and exit
--output_file OUTPUT_FILE
Full filepath of the output file.
--saved_model_dir SAVED_MODEL_DIR
Full path of the directory containing the SavedModel.
--keras_model_file KERAS_MODEL_FILE
Full filepath of HDF5 file containing tf.Keras model.
--saved_model_tag_set SAVED_MODEL_TAG_SET
Comma-separated set of tags identifying the MetaGraphDef within the SavedModel to analyze. All tags must be present. To pass in an empty
tag set, pass in "". (default "serve")
--saved_model_signature_key SAVED_MODEL_SIGNATURE_KEY
Key identifying the SignatureDef containing inputs and outputs. (default DEFAULT_SERVING_SIGNATURE_DEF_KEY)
--enable_v1_converter
Enables the TensorFlow V1 converter in 2.0
Copy to clipboard
### Convert a SavedModel
To convert a typical TensorFlow model in the saved\_model format using the tflite\_convert tool, run the following command:
tflite_convert \
--saved_model_dir=/tmp/mobilenet_saved_model \
--output_file=/tmp/mobilenet.tflite \
--saved_model_tag_set=serve \
--saved_model_signature_key="serving_default"
Copy to clipboard
### Convert a Keras H5 model
To convert a Keras model using the tflite\_convert tool, run the following command:
tflite_convert \
--keras_model_file=/tmp/mobilenet_keras_model.h5 \
--output_file=/tmp/mobilenet.tflite
Copy to clipboard
Note
The tflite\_convert tool is suitable for basic purposes only. For posttraining integer quantization, Qualcomm recommends using Python APIs.
## Create an application and run inference
You can use the LiteRT C++ APIs to create an application, load a LiteRT model, and
run it on hardware using delegates.
The following figure shows the steps involved in creating an application using C++ APIs to run a LiteRT model.
**Figure: Workflow to create an application and run a LiteRT model**
### Load a LiteRT model
A LiteRT model is a FlatBuffers file that has information on model operators and
any associated weights and biases.
The contents of the FlatBuffers file include the following:
- Tensors (input and outputs of each operation)
- Buffers (weights and biases)
- Operations that create an execution graph
The LiteRT framework provides APIs to do the following:
- Load a LiteRT model file
- Unpack all the content of the FlatBuffers file into memory
You can use the following APIs to load a LiteRT model for inference:
include
include
include "tensorflow/lite/interpreter.h"
include "tensorflow/lite/kernels/register.h"
include "tensorflow/lite/model.h"
include "tensorflow/lite/optional_debug_tools.h"
std::unique_ptr model;
model = tflite::FlatBufferModel::BuildFromFile(model_name.c_str());
if (!model) {
std::cerr << "Failed to mmap model " << model_name << std::endl;
exit(-1);
}
Copy to clipboard
### Create a LiteRT interpreter
Using the TensorFlow C/C++ APIs, you can build an interpreter to run the
model.
The interpreter interface helps you to do the following:
- Configure model execution on a chosen delegate.
- Assign the memory needed to for forward propagation.
The following example code demonstrates how you can create an interpreter. You can configure the interpreter instance to use a specific delegate and perform forward propagation.
//Build the interpreter with the InterpreterBuilder.
//Note: all Interpreters should be built with the InterpreterBuilder,
// which allocates memory for the Interpreter and does various set up
// tasks so that the Interpreter can read the provided model.
tflite::ops::builtin::BuiltinOpResolver resolver;
tflite::InterpreterBuilder builder(*model, resolver);
std::unique_ptr interpreter;
builder(&interpreter);
if (!interpreter) {
std::cerr << "Failed to construct interpreter on provided tflite model" << std::endl;
}
if (interpreter->AllocateTensors() != kTfLiteOk) {
std::cerr << "Failed to allocate tensors!" << std::endl;
exit(-1);
}
Copy to clipboard
### Prepare a model with a chosen delegate
After creating an interpreter and allocating the necessary memory to run the model,
prepare the model with a chosen delegate. This step creates an execution graph from the
model loaded earlier and uses the underlying library to perform inference on the delegate
hardware.
The following example code creates the XNNPACK delegate for running a LiteRT model on the Arm CPU. It creates the delegate by calling the `TfLiteXNNPackDelegateCreate(…)` API. You can also customize the delegate using the Delegate Options API.
TfLiteDelegate *delegate = NULL;
TfLiteXNNPackDelegateOptions xnnpack_options =
TfLiteXNNPackDelegateOptionsDefault();
xnnpack_options.num_threads = num_threads;
TfLiteDelegate* xnnpack_delegate =
TfLiteXNNPackDelegateCreate(&xnnpack_options);
if (interpreter->ModifyGraphWithDelegate(xnnpack_delegate) != kTfLiteOk) {
// Report error and fall back to another delegate, or the default backend
}
Copy to clipboard
### Prepare input/output buffers
When you build a standalone LiteRT application, it’s essential to prepare input data,
such as camera frames, for the pipeline to run LiteRT models.
Preprocessing operations, in the following cases for example, are important to ensure that inference happens correctly:
- Resizing the input image to a resolution expected by the model
- Normalization
- Mean subtraction
### Run a model
To run inference on a model, you must invoke a delegate using the
`Invoke()` API. Before invoking this API, create the appropriate
input/output buffers and provide them to the interpreter.
After the inference is complete, you can parse the output from the output buffers of the interpreter to generate the inference results.
An example of the `Invoke()` API running a model using a delegate is as follows:
// Run Inference
interpreter->Invoke()
Copy to clipboard
After the inference is complete, you can find the output tensors from the LiteRT `Invoke()` API in the output buffers of the interpreter. To perform further postprocessing on these outputs, you can parse them from the interpreter.
For a comprehensive example, see the label\_image example in the [TensorFlow GitHub repository](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/label_image).
For more information, see [LiteRT documentation](https://ai.google.dev/edge/litert).
## Develop a custom application
To enhance the developer experience, the Qualcomm IM SDK provides the qtimltflite
GStreamer-based plug-in, which performs LiteRT model inference.
For more details, see the following:
- [Qualcomm IM SDK](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-50/overview.html) documentation
- [qtimltflite](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-50/qtimltflite.html) plug-in documentation
- [Develop your own application](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-15B/develop-your-own-application.html)
Last Published: Apr 07, 2025
[Previous Topic
LiteRT architecture](https://docs.qualcomm.com/bundle/publicresource/80-70018-54/topics/arch.md) [Next Topic
Run LiteRT sample applications](https://docs.qualcomm.com/bundle/publicresource/80-70018-54/topics/sample-applications.md)