# TensorFlow Lite developer workflow
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
You can use an existing TensorFlow Lite model by downloading it from the open-source
community. Alternatively, you can convert a TensorFlow or Keras model to the TensorFlow Lite
format using specific tools. You can then run inference on a device and develop a custom
application for the TensorFlow Lite model.
Note: If you are using TensorFlow Lite models from Qualcomm AI Hub or
other sources, you may skip the tasks described in the TensorFlow Lite developer
workflow.
Executing a TensorFlow Lite model on Qualcomm-specific hardware involves the following
tasks.
## Convert a TensorFlow model to a TensorFlow Lite model
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
You can convert TensorFlow models to TensorFlow Lite models and optimize them for
on-device inference purposes. For more details on TensorFlow Lite model conversion, see
[Model conversion overview](https://www.tensorflow.org/lite/models/convert).
TensorFlow Lite model conversion supports converting models to the following formats:
- 32‑bit floating-point precision
- 16‑bit floating-point precision
- UINT8/INT8 precision (quantizing models)
### Use an existing TensorFlow Lite model
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
You can deploy an existing TensorFlow Lite model available in the open-source
community using TensorFlow Lite Runtime.
Qualcomm AI Hub publishes TensorFlow Lite models optimized to run on the Qualcomm Linux
Development Kit. For TensorFlow Lite models from Qualcomm, see [Qualcomm AI
Hub](https://aihub.qualcomm.com/).
To download an optimized model from Qualcomm AI Hub, do the following:
1. Go to [AI Hub Model Zoo](https://aihub.qualcomm.com/iot/models).
2. To filter the available models by chipset, select a chipset in the left pane.
3. Select a model.
4. On the next page, select the TorchScript
>
TFLite path.
5. Click Download model.
Note: The downloaded model is pre-optimized and ready for
deployment.
Figure : Optimized TensorFlow Lite model on Qualcomm AI Hub
### Convert a TensorFlow or Keras model to the TensorFlow Lite format
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
The TensorFlow framework provides Python APIs and a command-line interface (CLI) tool
to convert a TensorFlow or Keras model to the TensorFlow Lite format.
Table : TensorFlow model conversion methods
| Conversion method | Description |
| --- | --- |
| Python APIs | Converts, optimizes, and quantizes models to the TensorFlow Lite format |
| CLI tool | Converts models to the TensorFlow Lite format, but it is suitable for basic model conversion only |
Note: TensorFlow to TensorFlow Lite Python APIs offer more flexibility
to convert, optimize, and quantize models to suit your requirements.
### Convert and quantize using Python APIs
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
TensorFlow offers the following APIs to convert a TensorFlow SavedModel or a Keras
model to a TensorFlow Lite model:
Table : TensorFlow Python APIs to convert models
| API | Description |
| --- | --- |
| tf.lite.TFLiteConverter.from\_saved\_model() (recommended) | Converts a TensorFlow SavedModel |
| tf.lite.TFLiteConverter.from\_keras\_model() | Converts a Keras model |
### Convert a TensorFlow SavedModel (recommended)
The following example shows how to convert a TensorFlow model saved in the
saved\_model format to a TensorFlow Lite model:
import tensorflow as tf
# Convert the model
saved_model_dir = "/path/to/tf/model/in/saved_model/format"
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
# Save the model
with open("model.tflite", "wb") as f:
f.write(tflite_model)
Copy to clipboard
Note: The converted TensorFlow Lite model is not quantized and its
data is in 32‑bit floating-point precision.
### Convert a Keras model
The following example shows how to convert a Keras model to a TensorFlow Lite
model:
import tensorflow as tf
# Create a model using high-level tf.keras.* APIs
model = tf.keras.models.Sequential([
tf.keras.layers.Dense(units=1, input_shape=[1]),
tf.keras.layers.Dense(units=16, activation='relu'),
tf.keras.layers.Dense(units=1)
])
# compile the model
model.compile(optimizer='sgd', loss='mean_squared_error')
# train the model
model.fit(x=[-1, 0, 1], y=[-3, -1, 1], epochs=5)
# Convert the model to TFLite
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
# Save the model
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
Copy to clipboard
Note: The converted TensorFlow Lite model is not quantized and its
data is in 32‑bit floating-point precision.
### Quantize models
After converting a model to the TensorFlow Lite format, you can quantize it.
Quantization in neural network models involves the following steps:
1. Quantize weights and biases: Weights and biases are already part of the trained
model and can be quantized without additional information. Hence, quantizing
weights and biases is a static step.
2. Quantize activation layers: Ranges for the activation layer output depend on the
input image during forward propagation. Therefore, a set of sample inputs are
required to quantize these layers and identify the minimum/maximum ranges. Such
sample inputs are called calibration/representative data set.
To quantize a TensorFlow floating-point model to a quantized TensorFlow Lite model,
the TensorFlow Lite model provides posttraining quantization techniques. For more
information, see [Posttraining quantization](https://www.tensorflow.org/lite/performance/post_training_quantization).
### Posttraining quantization
TensorFlow Lite supports two types of posttraining quantizations:
- Posttraining dynamic range quantization
- Posttraining full-integer quantization
### Posttraining dynamic range quantization
In posttraining dynamic range quantization, weights and biases are quantized
statically from floating-point precision to fixed-point integer 8‑bit precision. The
activation layer ranges remain in 32‑bit floating-point precision.
To reduce latencies during inference, dynamic-range operators do the following:
- Quantize activations based on their ranges to fixed-point integer 8‑bit
precision
- Perform a computation with 8‑bit weights and activations
Note: No additional calibration data is needed in this step
because only weights are quantized.
The following script converts and quantizes a TensorFlow model to a TensorFlow Lite
model:
import tensorflow as tf
from tensorflow import keras
converter = tf.lite.TFLiteConverter.from_saved_model(exp_model_path)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
save_name = 'quantized_model.tflite'
print('Saving Dynamic Quantized TFLite model ..................')
with open(save_name, 'wb') as f:
f.write(tflite_model)
Copy to clipboard
### Posttraining full-integer quantization
In full-integer quantization, a representative data set is used to perform
quantization for activation layers within the model.
The following script converts and quantizes a TensorFlow model to a TensorFlow Lite
model. It generates a full-integer quantized model that is more suitable for a
fixed-point integer hardware such as the Hexagon Tensor Processor on the Qualcomm
Linux Development Kit.
import tensorflow as tf
def representative_dataset():
for data in dataset:
yield {
"image": data.image,
"bias": data.bias,
}
saved_model_dir = "/path/to/saved/model"
# prepare converter by loading model in saved_model format.
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# Set representative dataset used for quantization.
converter.representative_dataset = representative_dataset
# For full-integer quantization, set target_spec supported_ops to TFLITE_BUILTINS_INT8.
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8 # or tf.uint8
converter.inference_output_type = tf.int8 # or tf.uint8
# Convert model
tflite_quant_model = converter.convert()
save_name = 'quantized_model_int8.tflite'
print('Saving Quantized TFLite model ..................')
with open(save_name, 'wb') as f:
f.write(tflite_model)
Copy to clipboard
Note: `supported_ops` in converter
`target_spec` is set to
`tf.lite.OpsSet.TFLITE_BUILTINS_INT8`.
### Convert using offline converter tool (CLI)
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
The tflite\_convert TensorFlow Lite converter tool comes with the TensorFlow pip
package and can be used offline for TensorFlow versions 2.x and later.
The tflite\_convert tool accepts the following input in the CLI:
tflite_convert --help
optional arguments:
-h, --help show this help message and exit
--output_file OUTPUT_FILE
Full filepath of the output file.
--saved_model_dir SAVED_MODEL_DIR
Full path of the directory containing the SavedModel.
--keras_model_file KERAS_MODEL_FILE
Full filepath of HDF5 file containing tf.Keras model.
--saved_model_tag_set SAVED_MODEL_TAG_SET
Comma-separated set of tags identifying the MetaGraphDef within the SavedModel to analyze. All tags must be present. To pass in an empty
tag set, pass in "". (default "serve")
--saved_model_signature_key SAVED_MODEL_SIGNATURE_KEY
Key identifying the SignatureDef containing inputs and outputs. (default DEFAULT_SERVING_SIGNATURE_DEF_KEY)
--enable_v1_converter
Enables the TensorFlow V1 converter in 2.0
Copy to clipboard
### Convert a SavedModel
To convert a typical TensorFlow model contained in the saved\_model format using the
tflite\_convert tool, run the following command:
tflite_convert \ --saved_model_dir=/tmp/mobilenet_saved_model \ --output_file=/tmp/mobilenet.tflite \ --saved_model_tag_set=serve \
--saved_model_signature_key="serving_default"
Copy to clipboard
### Convert a Keras H5 model
To convert a Keras model using the tflite\_convert tool, run the following
command:
tflite_convert \
--keras_model_file=/tmp/mobilenet_keras_model.h5 \
--output_file=/tmp/mobilenet.tflite
Copy to clipboard
Note: The tflite\_convert tool is for basic purposes only. Python
APIs are recommended for posttraining integer quantization.
## Create an application and run inference
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
You can use the TensorFlow Lite C++ APIs to create an application, load a TensorFlow
Lite model, and execute the model on hardware using delegates.
A typical application created using C++ APIs to run a TensorFlow Lite model involves the
following steps:
Figure : Workflow to create an application and run a TensorFlow Lite model
### Load a TensorFlow Lite model
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
A TensorFlow Lite model is a FlatBuffers file that contains information on model
operators and any associated weights and biases.
The contents of the FlatBuffers file include the following:
- Tensors (input and outputs of each op)
- Buffers (weights and biases)
- Ops that create an execution graph
The TensorFlow Lite framework provides APIs to do the following:
- Load a TensorFlow Lite model file
- Unpack all the content of the FlatBuffers file onto the memory
You can use the following API to load a TensorFlow Lite model for inference:
include
include
include "tensorflow/lite/interpreter.h"
include "tensorflow/lite/kernels/register.h"
include "tensorflow/lite/model.h"
include "tensorflow/lite/optional_debug_tools.h"
std::unique_ptr model;
model = tflite::FlatBufferModel::BuildFromFile(model_name.c_str());
if (!model) {
std::cerr << "Failed to mmap model " << model_name << std::endl;
exit(-1);
}
Copy to clipboard
### Create a TensorFlow Lite interpreter
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
Using the TensorFlow C/C++ APIs, you can build an interpreter to run the
model.
The interpreter interface allows you to do the following:
- Configure model execution on a chosen delegate.
- Allocate the memory needed to perform forward propagation.
The following example code demonstrates how you can create an interpreter. You can
configure the interpreter instance to use a specific delegate and perform forward
propagation.
//Build the interpreter with the InterpreterBuilder.
//Note: all Interpreters should be built with the InterpreterBuilder,
// which allocates memory for the Interpreter and does various set up
// tasks so that the Interpreter can read the provided model.
tflite::ops::builtin::BuiltinOpResolver resolver;
tflite::InterpreterBuilder builder(*model, resolver);
std::unique_ptr interpreter;
builder(&interpreter);
if (!interpreter) {
std::cerr << "Failed to construct interpreter on provided tflite model" << std::endl;
}
if (interpreter->AllocateTensors() != kTfLiteOk) {
std::cerr << "Failed to allocate tensors!" << std::endl;
exit(-1);
}
Copy to clipboard
### Prepare a model with a chosen delegate
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
After creating an interpreter and allocating the necessary memory to run the model,
prepare the model with a chosen delegate. This step creates an execution graph from the
model loaded before and uses the underlying library to perform inference on the delegate
hardware.
The following example code creates the XNNPACK delegate for running a TensorFlow Lite
model on the Arm CPU. The delegate is created by calling the
`TfLiteXNNPackDelegateCreate(…)` API. You can also customize the
delegate with the Delegate Options API.
TfLiteDelegate *delegate = NULL;
TfLiteXNNPackDelegateOptions xnnpack_options =
TfLiteXNNPackDelegateOptionsDefault();
xnnpack_options.num_threads = num_threads;
TfLiteDelegate* xnnpack_delegate =
TfLiteXNNPackDelegateCreate(&xnnpack_options);
if (interpreter->ModifyGraphWithDelegate(xnnpack_delegate) != kTfLiteOk) {
// Report error and fall back to another delegate, or the default backend
}
Copy to clipboard
### Prepare input/output buffers
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
When building a standalone TensorFlow Lite application, the pipeline to execute
TensorFlow Lite models requires preparing input data; for example, camera
frames.
Preprocessing operations, in the following cases for example, are important to ensure
that inference is done correctly:
- Resizing the input image to a resolution that the model expects
- Normalization
- Mean subtraction
### Run a model
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
To run inference on a model, invoke a delegate using the `Invoke()`
API. Before invoking this API, create the appropriate input/output buffers and provide them
to the interpreter.
After inference is complete, you can parse the output from the interpreter output buffers
to get inference results.
An example of the `Invoke()` API executing a model using a delegate is as
follows:
// Run Inference
interpreter->Invoke()
Copy to clipboard
After inference is completed, output tensors from the TensorFlow Lite
`Invoke()` API are present in the output buffers of the interpreter.
To perform further postprocessing on these outputs, you can parse them from the
interpreter.
For a comprehensive example, see the label\_image example in the [TensorFlow GitHub repository](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/label_image).
For more information, see [TensorFlow Lite Guide](https://www.tensorflow.org/lite/guide).
## Develop a custom application
Source: [https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html](https://docs.qualcomm.com/doc/80-70015-54/topic/tensorflow-lite-developer-workflow.html)
To improve the developer experience, the IM SDK provides the qtimltflite
GStreamer-based plug-in, which performs TensorFlow Lite model inference.
For more details, see the following:
- [IM SDK](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-50/overview.html) documentation
- [qtimltflite](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-50/qtimltflite.html) plug-in documentation
- [Develop Your Own Application](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-15B/develop-own-app.html)
Last Published: Oct 09, 2024
[Previous Topic
Architecture](https://docs.qualcomm.com/bundle/publicresource/80-70015-54/topics/arch.md) [Next Topic
Sample applications](https://docs.qualcomm.com/bundle/publicresource/80-70015-54/topics/sample-applications.md)