# Run models
Tab Neural Processing Engine
Tab AI Engine Direct
- *class* tabincludedirective
- ## Deploy a model using Neural Processing Engine
The following video describes how to profile a model using Qualcomm Neural
Processing Engine.
A model DLC (quantized or non-quantized) can be deployed via a SNPE
enabled app (an application written using SNPE C/C++ APIs). SNPE
offers APIs to load a DLC, select a runtime to run the model, and
perform inference, etc.
SNPE provides a prebuilt
[snpe-net-run](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-2/tools.html#snpe-net-run)
tool (application written using C APIs) that can load an arbitrary
model DLC and run it on provided inputs.
- **Model file:** DLC model file generated by the SNPE converter tool
or `snpe-dlc-graph-prepare` tool (if running on HTP).
- **Input list:** Text file, like the `input_list.txt` file used during
quantization, except input raw files in this list are used for
inference. For simplicity in this example, the same input list used
for quantization is used for inference.
- **Runtime:** User must select a specific runtime to run the
model on target. Available runtime options are CPU, GPU, and DSP
(HTP).
Note
See `snpe-net-run --help` for more details.
### Run SNPE .dlc: x86 host computer
Converted SNPE .dlc can be run using the `snpe-net-run` tool which takes a
.dlc and input list as arguments. Running SNPE DLC on x86 is purely
for debugging purposes.
For example, the following command generates loads the `inception_v3.dlc`
model and runs the model on x86 CPU. It generates
output files to `/output_x86/`.
Note
The default runtime in SNPE is CPU. When executing a model
using `snpe-net-run`, there is no need to specify the CPU runtime.
${QAIRT_ROOT}/bin/x86_64-linux-clang/snpe-net-run --container ~/models/inception_v3.dlc --input_list ~/models/input_list.txt --output_dir ~/models/output_x86
Copy to clipboard
### Prepare a SNPE model to run on target
To run the model on target, `snpe-net-run` requires the model .dlc, SNPE
runtime libraries, and input list to run inferences to generate outputs.
Note
Before running on target, ensure that the SNPE SDK
binaries and libraries are pushed to the target.
Use artifacts from `${QAIRT_ROOT}/lib/aarch64-oe-linux-gcc11.2` and `${QAIRT_ROOT}/bin/aarch64-oe-linux-gcc11.2`
Use the correct DSP Hexagon architecture libraries for Qualcomm evaluation kits.
Tab IQ-8275
Tab IQ-9075
Tab Qualcomm Dragonwing™ RB3 Gen 2
Use v75 libraries
Use v73 libraries
Use v68 libraries
| **File** | **Source location** |
| --- | --- |
| snpe-net-run | ${QAIRT\_ROOT}/bin/aarch64-oe-linux-gcc11.2 |
| libSNPE.so | ${QAIRT\_ROOT}/lib/aarch64-oe-linux-gcc11.2/ |
| libSnpeHtpPrepare.so | ${QAIRT\_ROOT}/lib/aarch64-oe-linux-gcc11.2 |
| libSnpeHtpV68Stub.so | ${QAIRT\_ROOT}/lib/aarch64-oe-linux-gcc11.2 |
| libSnpeHtpV68Skel.so | ${QAIRT\_ROOT}/lib/hexagon-<dsp-arch>/unsigned |
| Inception\_v3\_quantized\_with\_htp\_cache.dlc | ~/models |
| Inception\_v3.dlc | ~/models |
| input\_list.txt | ~/models |
| Inception V3 sample input images | /tmp/RandomInputsForInceptionV3 |
The following scp commands needs to be executed on the host computer
to copy SNPE SDK libraries and binaries to the device.
Note
Replace `` with `75` for IQ-8275, `73` for IQ-9075, and `68` for Qualcomm Dragonwing™ RB3 Gen 2.
scp ${QAIRT_ROOT}/bin/aarch64-oe-linux-gcc11.2/snpe-net-run \
${QAIRT_ROOT}/lib/aarch64-oe-linux-gcc11.2/libSNPE.so \
${QAIRT_ROOT}/lib/aarch64-oe-linux-gcc11.2/libSnpeHtpVStub.so \
${QAIRT_ROOT}/lib/aarch64-oe-linux-gcc11.2/libSnpeHtpPrepare.so \
${QAIRT_ROOT}/lib/hexagon-v/unsigned/libSnpeHtpVSkel.so \
root@[ip-addr]:/opt/
Copy to clipboard
scp ~/models/inception_v3.dlc \
~/models/inception_v3_quantized_with_htp_cache.dlc \
~/models/input_list.txt \
root@[ip-addr]:/opt/
Copy to clipboard
scp -r /tmp/RandomInputsForInceptionV3 root@[ip-addr]:/tmp/
Copy to clipboard
ssh root@[ip-addr]
Copy to clipboard
cd /opt
Copy to clipboard
The above steps prepared the device for model execution. The
following sections provide details on running the model on the
available runtimes.
### Run SNPE .dlc: Arm-based CPU
Setup environment variables before executing the model to ensure SNPE
binaries and libraries are accessible to run the model.
Note
The default runtime in SNPE is CPU. When executing a model
using `snpe-net-run`, there is no need to specify the CPU runtime.
export LD_LIBRARY_PATH=/opt:$LD_LIBRARY_PATH
Copy to clipboard
export PATH=/opt/:$PATH
Copy to clipboard
snpe-net-run --container inception_v3.dlc --input_list input_list.txt --output_dir output_cpu
Copy to clipboard
### Run SNPE DLC: GPU
To run a model on the GPU runtime, use `inception_v3.dlc` and save outputs in `output_gpu`.
Note
To run on GPU, specify the runtime with the `--use_gpu` command line argument.
export LD_LIBRARY_PATH=/opt:$LD_LIBRARY_PATH
Copy to clipboard
export PATH=/opt:$PATH
Copy to clipboard
snpe-net-run --container inception_v3.dlc --input_list input_list.txt --output_dir output_gpu --use_gpu
Copy to clipboard
### Run SNPE DLC: HTP
To run a model on the HTP backend, use `inception_v3_quantized_with_htp_cache.dlc`
and save outputs in `output_htp`.
Note
To run on HTP, specify the runtime with the `--use_dsp` command line argument.
export LD_LIBRARY_PATH=/opt:$LD_LIBRARY_PATH
Copy to clipboard
export PATH=/opt:$PATH
Copy to clipboard
export ADSP_LIBRARY_PATH="/opt;/usr/lib/rfsa/adsp;/dsp"
Copy to clipboard
snpe-net-run --container inception_v3_quantized_with_htp_cache.dlc --input_list input_list.txt --output_dir output_htp --use_dsp
Copy to clipboard
### Validate output
For each input raw file fed to snpe-net-run (via the input\_list
file), an output folder is generated that contains output tensor
saved as raw file(s) whose size will match the model’s output layer
as shown in the image below.
Note
The [Netron](https://netron.app/) tool was used to visualize the model.
In the `inception_v3` example, the output raw file is a binary file
that contains probabilities for 1000 classification classes.
We use a Python script to read the file as a NumPy array to perform
postprocessing and output validation. The example below checks
whether HTP prediction is the same as CPU prediction.
1. Copy the output files from the target device to the host computer so
the outputs can be validated.
ssh root@[ip-addr]
Copy to clipboard
cd /opt/
Copy to clipboard
scp -r root@:/opt/output_cpu
Copy to clipboard
scp -r root@:/opt/output_htp
Copy to clipboard
Note
Use `` from the above commands in the following (`compare.py`) python script.
2. Once outputs from on-device inference are copied to the host computer,
prepare a script to load the output tensors saved in `output_htp`
and `output_cpu` to compare.
In this example, both `output_htp` and `output_cpu` are copied to
the `~/models` directory.
The following is an example script to compare output from one of the example
inputs used. Outputs are saved to the `/opt/` directory
on the host computer. Outputs from `snpe-net-run` can be loaded into
NumPy `ndarrays` using the `numpy.fromfile(…)` API.
Note
By default, `snpe-net-run` saves the output tensors to NumPy
files in **float32** format.
# python postprocessing script (compare.py) import numpy as np
import numpy as np
htp_output_file_path = "/output_htp/Result_1/875.raw"
cpu_output_file_path = "/output_cpu/Result_1/875.raw"
htp_output = np.fromfile(htp_output_file_path, dtype=np.float32)
htp_output = htp_output.reshape(1,1000)
cpu_output = np.fromfile(cpu_output_file_path, dtype=np.float32)
cpu_output = cpu_output.reshape(1,1000)
# np.argmax gives the cls_id with highest probability from tensor.
cls_id_htp = np.argmax(htp_output)
cls_id_cpu = np.argmax(cpu_output)
# Let's compare CPU output vs HTP output
print("Cpu prediction {} \n Htp Prediction {}".format(cls_id_cpu, cls_id_htp))
Copy to clipboard
Output
### Deployment using SNPE APIs
SNPE SDK provides C/C++ APIs to create/develop applications that run
a model on chosen hardware (CPU, GPU, or HTP) with acceleration. A
See the [sample application](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-2/cplus_plus_tutorial.html)
that demonstrates SNPE C/C++ APIs.
### Deployment using Qualcomm Intelligent Multimedia SDK (IM SDK)
To improve the developer experience when building entire use case
pipelines (stream from camera, preprocess images, perform inference,
etc.), SNPE has been integrated into the Qualcomm IM SDK as a plugin
[qtimlsnpe](https://docs.qualcomm.com/bundle/publicresource/topics/80-70023-50/qtimlsnpe.html).
The plugin has been developed on top of the SNPE C APIs and provides SNPE
capabilities (load and run models). With the `qtimlsnpe` plugin,
you can use your converted model DLC in a Qualcomm IM SDK
pipeline to realize the use case.
See [Develop your own AI/ML application](https://docs.qualcomm.com/doc/80-70023-15B/topic/develop-your-own-application.html)
for instructions on how to deploy SNPE DLC using the Qualcomm IM SDK.
- *class* tabincludedirective
- ## Deploy a model using AI Engine Direct
The following video describes how to profile a model using Qualcomm AI Engine Direct.
A model .so (quantized or non-quantized) can be deployed through a QNN
enabled app (an application written using QNN C/C++ APIs). QNN offers
APIs to load a model .so dynamically and run the model on hardware
with the selected backend.
QNN provides a prebuilt tool
([qnn-net-run](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/tools.html#qnn-net-run))
that can dynamically load this model .so and perform inference on a
selected backend using provided inputs.
For CPU, GPU, or HTP execution, `qnn-net-run` requires the following arguments:
- **Model file:** .so file generated by `qnn-model-lib-generator`
- **Backend file:** .so file for the targeted backend
>
>
> - `libQnnCpu.so` for the CPU backend.
> - `libQnnGpu.so` for the GPU backend.
> - `libQnnHtp.so` for the HTP backend.
- **Input list:** Text file like the input\_list.txt file used during
quantization, except input raw files in this list are used for
inference. For simplicity, this example uses the same input list
used for quantization for inference.
### Run QNN .so: x86 host computer
Converted QNN .so can be run using the `qnn-net-run` too which takes
a model .so, backend library .so, and input list as arguments.
For example, the following command loads the `libinception_v3.so`
model and runs the model on x86 CPU. It generates
output files to `~/models/output_x86/`.
${QAIRT_ROOT}/bin/x86_64-linux-clang/qnn-net-run --model ~/models/libs/x86_64-linux-clang/libinception_v3.so --backend ${QAIRT_ROOT}/lib/x86_64-linux-clang/libQnnCpu.so --input_list input_list.txt --output_dir ~/models/output_qnn_x86
Copy to clipboard
### Prepare a QNN model to run on target
To run the model on target, `qnn-net-run`, requires the model .so,
QNN binaries and runtime libraries, and input list to run inferences to generate outputs.
Note
Before running on target, ensure that the QNN SDK binaries and
runtime libraries are pushed to the target.
Use artifacts from `${QAIRT_ROOT}/bin/aarch64-oe-linux-gcc11.2` and `${QAIRT_ROOT}/lib/aarch64-oe-linux-gcc11.2`
Use the correct DSP Hexagon architecture libraries for Qualcomm evaluation kits.
Tab IQ-8275
Tab IQ-9075
Tab Qualcomm Dragonwing™ RB3 Gen 2
Use v75 libraries
Use v73 libraries.
Use v68 libraries.
| **File** | **Source location** |
| --- | --- |
| qnn-net-run | ${QAIRT\_ROOT}/bin/aarch64-oe-linux-gcc11.2 |
| libQnnHtp.so | ${QAIRT\_ROOT}/lib/aarch64-oe-linux-gcc11.2/libQnnHtp.so |
| libQnnCpu.so | ${QAIRT\_ROOT}/lib/aarch64-oe-linux-gcc11.2/libQnnCpu.so |
| libQnnGpu.so | ${QAIRT\_ROOT}/lib/aarch64-oe-linux-gcc11.2/libQnnGpu.so |
| libQnnHtpPrepare.so | ${QAIRT\_ROOT}/lib/aarch64-oe-linux-gcc11.2 |
| libQnnHtpV68Stub.so | ${QAIRT\_ROOT}/lib/aarch64-oe-linux-gcc11.2 |
| libinception\_v3\_quantized.s o | ~/models/libs/aarch64-oe-linux-gcc11.2 |
| libinception\_v3.so | ~/models/libs/aarch64-oe-linux-gcc11.2 |
| libQnnHtpV68Skel.so | ${QAIRT\_ROOT}/lib/hexagon-<dsp-arch>/unsigned |
| libqnnhtpv68.cat | ${QAIRT\_ROOT}/lib/hexagon-<dsp-arch>/unsigned |
| libQnnSaver.so | ${QAIRT\_ROOT}/lib/hexagon-<dsp-arch>/unsigned |
| libQnnSystem.so | ${QAIRT\_ROOT}/lib/hexagon-<dsp-arch>/unsigned |
The following `scp` commands need to be run on the host computer
to copy QNN SDK libraries and binaries to the device.
Note
Replace `` with `75` for IQ-8275, `73` for IQ-9075, and `68` for Qualcomm Dragonwing™ RB3 Gen 2.
scp ${QAIRT_ROOT}/bin/aarch64-oe-linux-gcc11.2/qnn-* \
${QAIRT_ROOT}/lib/aarch64-oe-linux-gcc11.2/libQnn*.so \
${QAIRT_ROOT}/lib/hexagon-v/unsigned/* \
root@[ip-addr]:/opt/
Copy to clipboard
scp ~/models/libs/aarch64-oe-linux-gcc11.2/* ~/models/input_list.txt root@[ip-addr]:/opt/
Copy to clipboard
scp -r /tmp/RandomInputsForInceptionV3 root@[ip-addr]:/tmp/
Copy to clipboard
ssh root@[ip-addr]
Copy to clipboard
cd /opt/
Copy to clipboard
The above steps prepared the device for model execution. The
following sections provide details on running the model on the
available runtimes.
### Run QNN .so: Arm-based CPU
When running a model on an Arm-based CPU, `qnn-net-run` requires the
model .so, backend .so library, and input list to run inference to generate outputs.
For RB3Gen2 targets, the .so must be cross-compiled with
the `aarch64-oe-linux-gcc11.2` toolchain)
For example, the following commands write output files
to `/opt/output_cpu/`.
export LD_LIBRARY_PATH=/opt/:$LD_LIBRARY_PATH
Copy to clipboard
export PATH=/opt:$PATH
Copy to clipboard
qnn-net-run --model libinception_v3.so --backend libQnnCpu.so --input_list input_list.txt --output_dir output_cpu
Copy to clipboard
### Run a QNN model on GPU
When executing a model on Adreno GPU, `qnn-net-run` requires the model .so,
backend .so library (`libQnnGpu.so`), and input list to run
inference to generate outputs.
For example, the following commands write output files
to `/opt/output_gpu/`.
export LD_LIBRARY_PATH=/opt/:$LD_LIBRARY_PATH
Copy to clipboard
export PATH=/opt:$PATH
Copy to clipboard
qnn-net-run --model libinception_v3.so --backend libQnnGpu.so --input_list input_list.txt --output_dir output_gpu
Copy to clipboard
### Run QNN model on HTP backend
To run a model on HTP backend, use the `libquantized_inception_v3.so`
library and `libQnnHtp.so` backend library and save outputs in the `output_htp`
directory.
export LD_LIBRARY_PATH=/opt/:$LD_LIBRARY_PATH
export PATH=/opt:$PATH
Copy to clipboard
export ADSP_LIBRARY_PATH="/opt/;/usr/lib/rfsa/adsp;/dsp"
Copy to clipboard
qnn-net-run --model libinception_v3_quantized.so --backend libQnnHtp.so --input_list input_list.txt --output_dir output_htp
Copy to clipboard
### Validate output
For each raw input file fed to `qnn-net-run` (through the input\_list file),
an output folder is generated that contains raw output file(s) whose
size matches the model’s output layer as shown in the image below.
In this `inception_v3` example, the raw output file is a binary file
that contains probability for 1000 classification classes.
You can use a Python script to read the file as a NumPy array to perform
postprocessing and validation of the output. The example script below checks
whether HTP prediction is the same as CPU prediction.
ssh root@[ip-addr]
Copy to clipboard
cd /opt/
Copy to clipboard
scp -r root@:/opt/output_cpu
Copy to clipboard
scp -r root@:/opt/output_htp
Copy to clipboard
Note
Use the `` from the above commands in the following Python script.
Once the outputs from the `qnn-net-run` tool are copied from the device
to the host computer, create a simple Python script to load outputs from
CPU and HTP execution and compare them using NumPy.
#python postprocessing script (compare.py)
import numpy as np
htp_output_file_path = "/output_htp/Result_1/_875.raw"
cpu_output_file_path = "/output_cpu/Result_1/_875.raw"
htp_output = np.fromfile(htp_output_file_path, dtype=np.float32)
htp_output = htp_output.reshape(1,1000)
cpu_output = np.fromfile(cpu_output_file_path, dtype=np.float32)
cpu_output = cpu_output.reshape(1,1000)
cls_id_htp = np.argmax(htp_output)
cls_id_cpu = np.argmax(cpu_output)
# Let's compare CPU output vs HTP output
print("CPU prediction {} \n HTP prediction {}".format(cls_id_cpu, cls_id_htp))
Copy to clipboard
Output
### Deployment using QNN APIs
The Qualcomm AI Engine Direct SDK provides C/C++ APIs to create/develop
applications that can load a compiled .so model and run it on a
chosen backend (CPU, GPU, or HTP) with acceleration. See the
[sample application](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/sample_app.html)
application that demonstrates QNN C/C++ APIs.
### Deployment using Qualcomm IM SDK
To improve the developer experience when building entire use case
pipelines (stream from camera, preprocess images, perform inference,
etc.), QNN has been integrated into the Qualcomm IM SDK as the
[qtimlqnn](https://docs.qualcomm.com/bundle/publicresource/topics/80-70023-50/qtimlqnn.html)
plugin.
The plugin has been developed on top of QNN C APIs and
provides capabilities to dynamically load and run models. With
the `qtimlqnn` plugin, developers can use their converted and compiled
model library in the Qualcomm IM SDK pipeline to realize their use case.
See [Develop your own AI/ML application](https://docs.qualcomm.com/doc/80-70023-15B/topic/develop-your-own-application.html)
for instructions on how to deploy QNN models using Qualcomm IM SDK.
Last Published: Dec 29, 2025
[Previous Topic
Port models](https://docs.qualcomm.com/bundle/publicresource/80-70023-15B/topics/port-models.md) [Next Topic
Fine tune an AI model with custom data using Edge Impulse](https://docs.qualcomm.com/bundle/publicresource/80-70023-15B/topics/edge-impulse.md)