# Execute the QPC After [compiling the model](https://docs.qualcomm.com/doc/80-99100-3/topic/index_model-compilation.html#reference-to-compile-the-model) into a QPC, the next step is to run inference on the device using the precompiled QPC. You can call inference on the Qualcomm Cloud AI 100 device in four ways: - Use `qaic-runner` - Use [Python APIs](https://docs.qualcomm.com/doc/80-99100-3/topic/index_Python-API.html#reference-to-python-api) - Use [C++ APIs](https://docs.qualcomm.com/doc/80-99100-3/topic/index_Cpp-API.html#reference-to-cpp-api) - Use [ONNX Runtime](https://docs.qualcomm.com/doc/80-99100-3/topic/index_onnxruntime.html) This section describes how to use `qaic-runner` to call inference, how to configure profiling options to collect [inference performance](https://docs.qualcomm.com/doc/80-99100-3/topic/index_model-execution.html#reference-to-inference-profiling) metrics, and how to run multiple models simultaneously on the same hardware using `qaic-program-group-app`. ## Use `qaic-runner` `qaic-runner` is a command line interface (CLI) tool designed to facilitate inference performance and benchmarking analysis. It provides various options and functionalities to facilitate inference and performance/benchmarking analysis. Use `qaic-runner` primarily for performance testing purposes. For actual inference tasks, it’s recommended to use the [Python APIs](https://docs.qualcomm.com/doc/80-99100-3/topic/index_Python-API.html#reference-to-python-api) or [C++ APIs](https://docs.qualcomm.com/doc/80-99100-3/topic/index_Cpp-API.html#reference-to-cpp-api), depending on your preferred technology stack. ### Prerequisites The examples in this section assume that a [QPC](https://docs.qualcomm.com/doc/80-99100-3/topic/index_model-compilation.html#reference-to-compile-the-model) is already generated. ### Usage examples The examples in this section show how to: - Run inference in `qaic-runner` with random inputs - Run inference on a set of inputs that you provide. - Generate dumps and device level profiling. #### Run inference in `qaic-runner` with random inputs This example shows running inference in `qaic-runner` with random inputs. Because inputs aren’t provided, you’re feeding a randomly generated input to the device with the appropriate dimensions and type inferred from the QPC. Use `qaic-runner` in this configuration to measure performance. sudo /opt/qti-aic/exec/qaic-runner -t /path/to/qpc -a 3 -n 5000 -d 0 -v Copy to clipboard - `-a 3`: Number of activations. Activations here refers to the number of instances of the network you want to run on the device. In this case 3 copies of the network can run in parallel on the device. > > > - If you assume each network was compiled with 4 cores, then the device needs at least 12 (3x4) cores free. > - You can use `sudo /opt/qti-aic/tools/qaic-qpc validate -i /path/to/qpc/programqpc.bin` and look for `Number of NSP required` in the output to find the number of cores. - `-n 5000`: Number of iterations. This single randomly generated input is used for 5000 inferences. - `-d 0`: The device ID. Use `/opt/qti-aic/tools/qaic-util -q` to find the device ID. - `-v`: Enables verbose logging. #### Run inference on a set of inputs Before running inference, it’s necessary to convert the inputs to the appropriate format based on input size and type. Look at these [Jupyter notebook examples](https://github.com/quic/cloud-ai-sdk/blob/1.10/tutorials/NLP/Model-Onboarding-Beginner). #### Generate dumps for latency capture The following example command shows how to generate latency stats: !/opt/qti-aic/exec/qaic-runner -t ./BERT_LARGE -a 8 -S 1 -d 0 \ #-i inputFiles/input.raw \ --aic-profiling-format latency --aic-profiling-out-dir ./BERT_LARGE_STATS \ --aic-profiling-start-iter 100 --aic-profiling-num-samples 99999 --time 20 Copy to clipboard - `aic-profiling-out-dir` : Output directory for latency capture (needs to exist before this command is run). - `aic-profiling-start-iter` : Set this value high enough to start capturing samples after device warmup. - `aic-profiling-num-samples` : Number of samples to be captured. Can be set greater than the number of inferences. Look at this Jupyter notebook [example](https://github.com/quic/cloud-ai-sdk/tree/main/tutorials/NLP/Profiler-Intermediate). ### Parameters and default values The following table lists the `qaic-runner` parameters and default values:

Parameter

Description

Default

-d, --aic-device-id <id>

Specify AIC device ID.

0

-D, --dev-list <qid>[:<qid>]

Map of device IDs for a multi-device network.

0[:1]

-d, --aic-device-id <id>

AIC device ID, default Auto-pick

0

-D, --aic-device-map <qid>[:<qid>]

Map of Device IDs for multi-device network,

0[:1]

-t, --test-data <path>

Location of program binaries

-i, --input-file <path>

Input filename from which to load input data. Specify multiple times for each input file. If no -i is specified, look for available inputs in bindings.json at the -t directory. If bindings.json isn’t available, random input will be generated.

-n, --num-iter <num>

Number of iterations,

40

--time <t>

Duration (in seconds) for which to submit inferences

-l, --live-reporting

Enable Live reporting periodic at 1 sec interval

off

-r, --live-reporting-period

Set Live Reporting Period in Ms

1000

-s --stats

Enable Live Profiling Stats reporting periodically at 1 sec interval

-a, --aic-num-of-activations <num>

Number of activations

1

--aic-profiling-start-iter <num>

Profiling Start Iteration

0

--aic-profiling-start-delay <num>

Profiling Start delay (in milliseconds). Profiling will start after the specified delay period has elapsed

--aic-profiling-num-samples <num>

Profiling Num Samples to save to file

1

--aic-profiling-format <level>

Deprecated

DEF

--aic-profiling-type <type>

Profiling Type, ‘stats’|’trace’|’latency’ for legacy profiling and ‘trace_stream’ | ‘latency_stream’ for stream profiling. Set multiple times for multiple formats

none

--aic-profiling-duration <num>

Profiling duration to run profiling for (in ms). After starting profiling, it will stop at the expiry of profiling duration

--aic-profiling-sampling-rate <num>

Profiling sampling rate [full/half/fourth/eighth/sixteenth]. Programs generate profiling samples at the requested rate. Select full to profile all samples or select a fractional rate to profile samples at that interval

--aic-profiling-reporting-rate <num>

Profiling report generation rate (in ms) [500/1000/2000/4000]. Profiling report will be generated at every requested interval for profiling duration

500

--aic-profiling-out-dir <path>

Location to save files, dir should exist and be writable

‘.’

--write-output-start-iter <num>

Write outputs start iteration

0

--write-output-num-samples <num>

Number of outputs to write

1

--write-output-dir <path>

Location to save output files, dir should exist and be writable

‘.’

--aic-lib-path DEPRECATED

Deprecated, please set env variable QAIC_LIB to the full path of the custom library, by default loads libQAic.so from install location

--aic-batch-input-directory

Batch mode: process all files from input directory. Only the networks with single input file are currently supported

DEF

--aic-batch-input-file-list

Batch mode: Specify an input file containing comma-separated absolute path for buffers. Each line is 1 inference and must have number and size of buffers required by program

DEF

--aic-batch-max-memory <mb>

Batch mode: Limit memory usage when loading files, provide parameter in Mb

1024

--submit-timeout <num>

Time to wait for an inference request completion on kernel. default 0 ms. When 0, kernel defaults to 5000ms

--submit-retry-count <num>

Number of wait-call retries when an inference request times out.

5

--unbound-random

When populating random values in buffer, don’t consider input buffer format and fill each byte with random input between 0 to 255. This can result in unexpected behavior from certain network.

--dump-input-buffers

Dump input buffers used in benchmarking mode

-S, --set-size <num>

Set Size for inference loop execution, min:1

10

-T, --aic-threads-per-queue

Number of threads per queue

4

--auto-batch-input

Automatically batch inputs to meet batchsize requirements of network. Inputs should be for Batch size 1

1

-p, --pre-postprocessing

Pre-post processing [on|off]

on

-v, --verbose

Verbose log from program

-h, --help

help
See [QAic runner](https://docs.qualcomm.com/doc/80-99100-3/topic/qaic_runner.html) for more examples and a complete list of `qaic-runner` arguments. ## Profile inference time Cloud AI supports both system-level and device-level profiling to help you identify performance bottlenecks. Device-level profiling requires that you compile the model with the appropriate stats-level flags before running inference. See [Compile-time profiling options](https://docs.qualcomm.com/doc/80-99100-3/topic/index_model-compilation.html#reference-to-compile-time-profiling-options) for details. The complete device-level profiling workflow involves three steps: 1. Compile the model with profiling flags using `qaic-compile`. 2. Run inference and collect stats buffers using `qaic-runner`. 3. Post-process the stats buffers using `qaic-opstats`. The following sections describe the system-level and device-level profiling options available at runtime (Steps 2 and 3). See [Compile-time profiling options](https://docs.qualcomm.com/doc/80-99100-3/topic/index_model-compilation.html#reference-to-compile-time-profiling-options) for Step 1. ### Profile the system System-level profiling includes the breakdown of the inference time between application, linux runtime, kernel mode driver and device processing. This profiling can provide insights into the inference time spent on the host vs. device per inference. You can optimize your application or the model with this information. See the [Profiler notebook](https://github.com/quic/cloud-ai-sdk/tree/1.20/tutorials/NLP/Profiler-Intermediate) for an example of the complete system-level profiling workflow using `qaic-runner`. Profiling using the [Python APIs](https://docs.qualcomm.com/doc/80-99100-3/topic/index_Python-API.html#reference-to-python-api) is also supported. ### Profile the device Device level profiling identifies bottlenecks in the inference execution on the device. This profiling requires a good understanding of the AI core and SoC architecture. Three key features are: - Memory metrics: Provides a compiler estimate of the usage of on-board DDR vs. VTCM (vector tightly coupled memory) for a model. - Summary view: Provides a histogram of the operations, total time taken by every operation, where the operands are stored (DDR vs VTCM), effective usage of the individual IP blocks in the AI cores. Use for debug only because it can impact performance based on the size of the model. - Timeline view: Provides a timeline view of all the operations running across all IP blocks from start of an inference till the end. Used primarily to zoom into the operations to understand bottlenecks. See the [Profiler notebooks](https://github.com/quic/cloud-ai-sdk/tree/1.20/tutorials/NLP/Profiler-Intermediate) for an example of the complete device level profiling workflow for a model using `qaic-compile` and `qaic-runner`. ## Next steps - To use Python APIs to start the QPC and run inference, see [Python API](https://docs.qualcomm.com/doc/80-99100-3/topic/index_Python-API.html#reference-to-python-api). - To use C++ APIs to start the QPC and run inference, see [C++ API](https://docs.qualcomm.com/doc/80-99100-3/topic/index_Cpp-API.html#reference-to-cpp-api). - To use ONNX Runtime to start the QPC and run inference, see [using Qualcomm Cloud AI 100 as execution provider in ONNX Runtime](https://docs.qualcomm.com/doc/80-99100-3/topic/index_onnxruntime.html). Last Published: Aug 25, 2026 [Previous Topic Example run and img2raw usage for a ResNet-50 model](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/example-run-and-img2raw-usage-for-a-resnet-50-model.md) [Next Topic QAic runner](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/qaic_runner.md)