# Deploy a LiteRT model
You can use an existing LiteRT model by downloading it from the open-source community or convert a TensorFlow model or a Keras model to the LiteRT format using specific tools. After converting the model, you can create an application and run inference on a device, and develop a custom application for the LiteRT model.
The following figure shows you can either use pre-optimized models or convert models to LiteRT models using Python APIs and then quantize them to run inference. You can also use the `tflite_convert` command for basic model conversion.
**Figure: LiteRT model usage workflow**
## Use a pre-optimized LiteRT model
You can deploy an existing LiteRT model from the open-source community using LiteRT. The downloaded model is pre-optimized and ready for deployment. Qualcomm AI Hub publishes LiteRT models optimized for Qualcomm development kits. For LiteRT models from Qualcomm, see [Qualcomm AI Hub](https://aihub.qualcomm.com/).
To download a pre-optimized model from Qualcomm AI Hub, do the following:
1. Go to the [Qualcomm AI Hub IOT models](https://aihub.qualcomm.com/iot/models) page.
2. In the left pane, choose a chipset.
3. Choose a model.
4. Select Download Model.
5. In the Download Model dialog, choose runtime, precision, and device.
6. Select Download model.
**Figure: Pre-optimized LiteRT model on Qualcomm AI Hub**
Note
If you are using LiteRT models from Qualcomm AI Hub or other sources, you can skip model conversion and directly create an application to run inference. Model conversion requires a thorough understanding of the TensorFlow and LiteRT frameworks.
## Convert a TensorFlow model to a LiteRT model
You can convert TensorFlow models to LiteRT models and optimize them for on-device inference. For more information about LiteRT model conversion, see [Model conversion overview](https://ai.google.dev/edge/litert/models/convert).
LiteRT model conversion supports converting models to the following formats:
- 32‑bit floating-point precision
- 16‑bit floating-point precision
- uint8/int8 precision (quantizing models)
The following table lists the conversion methods, which the TensorFlow framework provides to convert a TensorFlow model or a Keras model to the LiteRT format:
Table: TensorFlow model conversion methods
| Conversion method | Description |
| --- | --- |
| Python APIs | Converts, optimizes, and quantizes models to the LiteRT format |
| Command-line interface (CLI) tool | Converts models to the LiteRT format, but is suitable for basic model conversion only |
The TensorFlow to LiteRT Python APIs offer more flexibility to convert, optimize, and quantize models to suit your requirements.
### Convert models using Python APIs
The following table lists the Python APIs that TensorFlow provides to convert a TensorFlow SavedModel or a Keras model to a LiteRT model:
Table: TensorFlow Python APIs to convert models
| API | Description |
| --- | --- |
| `tf.lite.TFLiteConverter.from_saved_model()` (recommended) | Converts a TensorFlow SavedModel |
| `tf.lite.TFLiteConverter.from_keras_model()` | Converts a Keras model |
#### Recommended: Convert a TensorFlow SavedModel using the Python API
The following example shows how to convert a TensorFlow model saved in the saved\_model format to a LiteRT model:
import tensorflow as tf
# Convert the model
saved_model_dir = "/path/to/tf/model/in/saved_model/format"
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
# Save the model
with open("model.tflite", "wb") as f:
f.write(tflite_model)
Copy to clipboard
Note
The converted LiteRT model isn’t quantized, and its data is in 32‑bit floating-point precision.
#### Convert a Keras model using the Python API
The following example shows how to convert a Keras model to a LiteRT model:
import tensorflow as tf
# Create a model using high-level tf.keras.* APIs
model = tf.keras.models.Sequential([
tf.keras.layers.Dense(units=1, input_shape=[1]),
tf.keras.layers.Dense(units=16, activation='relu'),
tf.keras.layers.Dense(units=1)
])
# compile the model
model.compile(optimizer='sgd', loss='mean_squared_error')
# train the model
model.fit(x=[-1, 0, 1], y=[-3, -1, 1], epochs=5)
# Convert the model to LiteRT
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
# Save the model
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
Copy to clipboard
Note
The converted LiteRT model isn’t quantized, and its data is in 32‑bit floating-point precision.
#### Quantize models
>
>
> After converting models to the LiteRT format using Python APIs, you can quantize them. Quantization reduces the size and computational requirements of models. It involves converting high-precision values, such as 32-bit floating-point numbers, into lower-precision formats, such as 8-bit integers.
Quantization in neural network models involves the following steps:
1. Quantize weights and biases: These are already part of the trained model; you can quantize them without additional information. Therefore, quantizing weights and biases is a static step.
2. Quantize activation layers: The ranges for the activation layer output depend on the input image during forward propagation. Therefore, a set of sample inputs, known as calibration or representative data sets, is necessary to quantize these layers and identify the minimum and maximum ranges.
To quantize a TensorFlow floating-point model to a quantized LiteRT model, LiteRT provides posttraining quantization techniques. For more information, see [Posttraining quantization](https://ai.google.dev/edge/litert/models/post_training_quantization).
LiteRT supports the following types of posttraining quantizations:
- Dynamic range quantization
- Full integer quantization
##### Quantize models using dynamic range quantization
In dynamic range quantization, weights and biases are statically quantized from floating-point precision to fixed-point integer 8‑bit precision. The activation layer ranges remain in 32‑bit floating-point precision.
To reduce latencies during inference, dynamic-range operators do the following:
- Quantize activations based on their ranges to fixed-point integer 8‑bit precision
- Perform computations with 8‑bit weights and activations
Note
This step only quantizes weights and doesn’t need extra calibration data.
The following script converts a TensorFlow model to a LiteRT model and then quantizes it:
import tensorflow as tf
from tensorflow import keras
converter = tf.lite.TFLiteConverter.from_saved_model(exp_model_path)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
save_name = 'quantized_model.tflite'
print('Saving Dynamic Quantized LiteRT model ..................')
with open(save_name, 'wb') as f:
f.write(tflite_model)
Copy to clipboard
##### Quantize models using full integer quantization
In full integer quantization, a representative data quantizes the activation layers within the model.
The following script converts and quantizes a TensorFlow model to a LiteRT model. It generates a full integer quantized model that’s more suitable for fixed-point integer hardware, such as the Hexagon Tensor Processor on Qualcomm development kits.
import tensorflow as tf
def representative_dataset():
for data in dataset:
yield {
"image": data.image,
"bias": data.bias,
}
saved_model_dir = "/path/to/saved/model"
# prepare converter by loading model in saved_model format.
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# Set representative dataset used for quantization.
converter.representative_dataset = representative_dataset
# For full-integer quantization, set target_spec supported_ops to TFLITE_BUILTINS_INT8.
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8 # or tf.uint8
converter.inference_output_type = tf.int8 # or tf.uint8
# Convert model
tflite_quant_model = converter.convert()
save_name = 'quantized_model_int8.tflite'
print('Saving Quantized LiteRT model ..................')
with open(save_name, 'wb') as f:
f.write(tflite_model)
Copy to clipboard
Note
`supported_ops` in converter sets `target_spec` to `tf.lite.OpsSet.TFLITE_BUILTINS_INT8`.
### Convert models using the `tflite_convert` command
You can use the TensorFlow pip package, which includes the `tflite_convert` TensorFlow Lite offline converter tool (CLI), for offline conversions with TensorFlow v2.x and later.
The `tflite_convert` command accepts the following input in the CLI:
tflite_convert --help
`--output_file`. Type: string. Full path of the output file.
`--saved_model_dir`. Type: string. Full path to the SavedModel directory.
`--keras_model_file`. Type: string. Full path to the Keras H5 model file.
`--enable_v1_converter`. Type: bool. (default False) Enables the converter and flags used in TF 1.x instead of TF 2.x.
You are required to provide the `--output_file` flag and either the `--saved_model_dir` or `--keras_model_file` flag.
Copy to clipboard
You can convert the following models using the `tflite_convert` command:
- SavedModel
- Keras H5
#### Convert a SavedModel using the `tflite_convert` command
To convert a typical TensorFlow model in the saved\_model format using the `tflite_convert` command, run the following command:
tflite_convert \
--saved_model_dir=/tmp/mobilenet_saved_model \
--output_file=/tmp/mobilenet.tflite
Copy to clipboard
#### Convert a Keras H5 model using the `tflite_convert` command
To convert a Keras H5 model using the `tflite_convert` command, run the following command:
tflite_convert \
--keras_model_file=/tmp/mobilenet_keras_model.h5 \
--output_file=/tmp/mobilenet.tflite
Copy to clipboard
Note
The `tflite_convert` command is suitable for basic purposes only. For posttraining integer quantization, use Python APIs.
## Create an application and run inference
After converting a TensorFlow model or a Keras model to a LiteRT model, you can use the LiteRT C++ APIs to create an application, load the LiteRT model, and run it on the hardware using delegates.
The following figure shows the steps involved in creating an application using C++ APIs to run a LiteRT model:
**Figure: Workflow to create an application and run a LiteRT model**
### Load a LiteRT model
A LiteRT model is a FlatBuffers file that has information on model operators and
any associated weights and biases.
The contents of the FlatBuffers file include the following:
- Tensors (input and outputs of each operation)
- Buffers (weights and biases)
- Operations that create an execution graph
The LiteRT framework provides APIs to do the following:
- Load a LiteRT model file
- Unpack all the content of the FlatBuffers file into memory
Use the following APIs to load a LiteRT model for inference:
include
include
include "tensorflow/lite/interpreter.h"
include "tensorflow/lite/kernels/register.h"
include "tensorflow/lite/model.h"
include "tensorflow/lite/optional_debug_tools.h"
std::unique_ptr model;
model = tflite::FlatBufferModel::BuildFromFile(model_name.c_str());
if (!model) {
std::cerr << "Failed to mmap model " << model_name << std::endl;
exit(-1);
}
Copy to clipboard
### Create a LiteRT interpreter
Using the TensorFlow C/C++ APIs, you can build an interpreter to run the
model.
The interpreter interface helps you to do the following:
- Configure model execution on a chosen delegate.
- Assign the memory needed for forward propagation.
The following example code demonstrates how you can create an interpreter. You can configure the interpreter instance to use a specific delegate and perform forward propagation.
//Build the interpreter with the InterpreterBuilder.
//Note: all Interpreters should be built with the InterpreterBuilder,
// which allocates memory for the Interpreter and does various set up
// tasks so that the Interpreter can read the provided model.
tflite::ops::builtin::BuiltinOpResolver resolver;
tflite::InterpreterBuilder builder(*model, resolver);
std::unique_ptr interpreter;
builder(&interpreter);
if (!interpreter) {
std::cerr << "Failed to construct interpreter on provided tflite model" << std::endl;
}
if (interpreter->AllocateTensors() != kTfLiteOk) {
std::cerr << "Failed to allocate tensors!" << std::endl;
exit(-1);
}
Copy to clipboard
### Prepare a model with a chosen delegate
After creating an interpreter and allocating the necessary memory to run the model,
prepare the model with a chosen delegate. This step creates an execution graph from the
model loaded earlier and uses the underlying library to perform inference on the delegate
hardware.
The following example code creates the XNNPACK delegate for running a LiteRT model on the Arm CPU. It creates the delegate by calling the `TfLiteXNNPackDelegateCreate(…)` API. You can also customize the delegate using the Delegate Options API.
TfLiteDelegate *delegate = NULL;
TfLiteXNNPackDelegateOptions xnnpack_options =
TfLiteXNNPackDelegateOptionsDefault();
xnnpack_options.num_threads = num_threads;
TfLiteDelegate* xnnpack_delegate =
TfLiteXNNPackDelegateCreate(&xnnpack_options);
if (interpreter->ModifyGraphWithDelegate(xnnpack_delegate) != kTfLiteOk) {
// Report error and fall back to another delegate, or the default backend
}
Copy to clipboard
### Prepare input/output buffers
When you build a standalone LiteRT application, it’s essential to prepare input data,
such as camera frames, for the pipeline to run LiteRT models.
Preprocessing operations, in the following cases for example, are important to ensure that inference happens correctly:
- Resizing the input image to a resolution expected by the model
- Normalization
- Mean subtraction
### Run a model
To run inference on a model, you must invoke a delegate using the
`Invoke()` API. Before invoking this API, create the appropriate
input/output buffers and provide them to the interpreter.
After the inference is complete, you can parse the output from the output buffers of the interpreter to generate the inference results.
An example of the `Invoke()` API running a model using a delegate is as follows:
// Run Inference
interpreter->Invoke()
Copy to clipboard
After the inference is complete, you can find the output tensors from the LiteRT `Invoke()` API in the output buffers of the interpreter. To perform further postprocessing on these outputs, you can parse them from the interpreter.
For a comprehensive example, see the label\_image example in the [TensorFlow GitHub repository](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/label_image).
For more information, see [LiteRT documentation](https://ai.google.dev/edge/litert).
## Develop a custom application
To enhance developer experience, the Qualcomm IM SDK provides the qtimltflite
GStreamer-based plug-in, which performs LiteRT model inference.
For more information, see the following:
- [Qualcomm IM SDK](https://docs.qualcomm.com/bundle/publicresource/topics/80-70029-50) documentation
- [qtimltflite](https://docs.qualcomm.com/bundle/publicresource/topics/80-70029-50/qtimltflite.html) plug-in documentation
- [Develop your own AI/ML application](https://docs.qualcomm.com/bundle/publicresource/topics/80-70029-15B/develop-your-own-application.html)
- [Qualcomm IM SDK](https://docs.qualcomm.com/bundle/publicresource/topics/80-70029-50) documentation
- [qtimltflite](https://docs.qualcomm.com/bundle/publicresource/topics/80-70029-50/qtimltflite.html) plug-in documentation
- [Develop your own AI/ML application](https://docs.qualcomm.com/bundle/publicresource/topics/80-90441-15/develop-your-own-application.html)
## Next steps
- Run LiteRT sample applications
Last Published: Mar 30, 2026
Previous Topic
LiteRT architecture Next Topic
Run LiteRT sample applications