# Develop an AI application using QAIRT C++ APIs

Note

Support for support for this section on Ubuntu will be available soon.

The Qualcomm AI Runtime (QAIRT) SDK provides C++ APIs for sample
application development. Samples are available for both Qualcomm AI Engine Direct (QNN)
and Qualcomm Neural Processing Engine SDK (SNPE). The samples help you begin application
development. The following instructions describe how to build, run, and navigate the
source code. They demonstrate the workflow for utilizing QNN or SNPE APIs to run a model.

## Build and run the QNN sample app

The `qnn-sample-app` is located at `${QNN_SDK_ROOT}/examples/QNN/SampleApp`,
where `QNN_SDK_ROOT` refers to the path where the QNN SDK has been extracted.

### Set up the QAIRT SDK

To setup the toolchain for the QNN sample app, do the following:

1. [Download the Qualcomm AI Runtime SDK](https://softwarecenter.qualcomm.com/api/download/software/sdks/Qualcomm_AI_Runtime_Community/All/2.43.0.260128/v2.43.0.260128.zip).
2. Extract and unzip the SDK.

unzip v2.43.0.260128.zip
        Copy to clipboard

cd qairt/2.43.0.260128
        Copy to clipboard

export QNN_SDK_ROOT=`pwd`
        Copy to clipboard
3. Install the eSDK.

    Follow the [Qualcomm IM SDK quickstart](https://docs.qualcomm.com/doc/80-70022-51/topic/install-sdk.html#section-b5c-z3k-5bc) to install the eSDK, which contains the
required cross-compiler toolchain.

    - For Yocto Scarthgap devices, the libraries are compiled with GCC-11.2.
    - Set the `ESDK_PATH` environment variable with eSDK installation path. Later steps use the installation path (`/path/to/extracted/toolchain`) for the compilation.

export ESDK_PATH="/path/to/extracted/toolchain"
            Copy to clipboard

### Build the QNN sample app

Follow the steps below to setup the toolchain for the QNN sample app.

1. Go to the sample app directory.

cd ${QNN_SDK_ROOT}/examples/QNN/SampleApp/SampleApp/
        Copy to clipboard
2. Set the environment variable for the GCC toolchain.

export QNN_AARCH64_LINUX_OE_GCC_112=$ESDK_PATH
        Copy to clipboard
3. Build the application.

make CXX="$ESDK_PATH/tmp/sysroots/x86_64/usr/bin/aarch64-qcom-linux/aarch64-qcom-linux-g++ --sysroot=$ESDK_PATH/tmp/sysroots/qcs6490-rb3gen2-vision-kit/" all_linux_oe_aarch64_gcc112
        Copy to clipboard

    This creates two folders.

    - `bin`: Contains `qnn-sample-app` binaries for each platform within their respective directories.
    - `obj`: Contains all object files used in building and linking the executable.

### Run the QNN sample app on Linux (Yocto-based)

The built `qnn-sample-app` executable can run a model with any
QNN backend. For Yocto scarthgap-based devices, backends are available
for `aarch64-oe-linux-gcc11.2`.

1. Push the artifacts to the target device.

scp ${QNN_SDK_ROOT}/examples/QNN/SampleApp/SampleApp/bin/aarch64-oe-linux-gcc11.2/qnn-sample-app root@[ip-addr]:/etc/apps/qnn-sample-app
        Copy to clipboard

Note

Create the `/etc/apps/` directory if it doesn’t already exist on the device.
2. On the host computer, use [AI Hub](https://docs.qualcomm.com/doc/80-70029-15B/topic/ai-hub.html) to export a model.

    For example, to export the InceptionV3 QNN model, run the following commands:

pip3 install qai-hub-models
        Copy to clipboard

python -m qai_hub_models.models.inception_v3.export --quantize w8a8 --target-runtime=qnn_context_binary --chipset="qualcomm-qcs6490-proxy" --compile-options="--qairt_version 2.40" --profile-options "--qairt_version 2.40"
        Copy to clipboard

Note

Generate the context binary for the same SDK version in use on the target device.
3. Push the exported InceptionV3 QNN model to the target device.

scp build/inception_v3_w8a8/inception_v3_w8a8.bin root@<ip-addr>:/etc/apps/
        Copy to clipboard

    When prompted to enter the password, enter oelinux123.
4. From the host computer, SSH into the target device.

ssh root@<IP_ADDRESS_OF_TARGET_DEVICE>
        Copy to clipboard
5. Generate a dummy input file.

cd /etc/apps
        Copy to clipboard

python3
        Copy to clipboard

    1. Run the following commands in the Python environment.

import numpy as np
            Copy to clipboard

((np.random.random((1,3,224,224)).astype(np.float32))).tofile("input.raw")
            Copy to clipboard
6. Create `input_list.txt`.

echo "input.raw" > /etc/apps/input_list.txt
        Copy to clipboard
7. Run the app.

chmod +x qnn-sample-app
        Copy to clipboard

./qnn-sample-app --retrieve_context inception_v3_w8a8.bin --backend libQnnHtp.so --input_list input_list.txt --system_library libQnnSystem.so
        Copy to clipboard

Note

Update the model name and input\_list as per the selected model.

    For help context, run:

./qnn-sample-app --help
        Copy to clipboard

Command line arguments

- **Required arguments**
    - - `--model`: Path to the QNN network model. Mutually exclusive with `--retrieve_context`.
- `--retrieve_context`: Path to a cached binary for loading a saved
context and execution graphs. Mutually exclusive with `--model`.
- `--backend`: Path to a QNN backend to run the model.
- `--input_list`: Path to a file listing network inputs. For multiple
graphs, provide a comma-separated list of input files.

- **Optional arguments**
    - - `--debug`: Save output from all network layers.
- `--output_dir`: Directory for outputs (default: ./output).
- `--output_data_type`: Output data type (float\_only, native\_only, float\_and\_native).
- `--input_data_type`: Input data type (float or native).
- `--op_packages`: Comma-separated list of op packages and interface providers.
- `--profiling_level`: Profiling level (basic or detailed).
- `--save_context`: Save backend context and graph metadata to a binary file.
- `--num_inferences`: Number of inferences to perform.
- `--log_level`: Max logging level (error, warn, info, verbose).
- `--system_library`: Path to libQnnSystem.so for reflection APIs during context loading.
- `--version`: Print QNN SDK version.
- `--help`: Display help message.

## Workflow and API usage

Use the following recommended pattern to develop C++ applications using QNN APIs.

1. [Load prerequisite shared libraries.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#loading-pre-requisite-shared-libraries)
2. [Use QNN APIs.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#usage-of-qnn-apis)

    1. [Use QNN interface to obtain function pointers.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#use-qnn-interface-to-obtain-function-pointers)
    2. [Set up logging.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#set-up-logging)
    3. [Initialize backend.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#initialize-backend)
    4. [Initialize profiling.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#initialize-profiling)
    5. [Create device.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#create-device)
    6. [Register op packages.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#register-op-packages)
    7. [Create context.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#create-context)
    8. [Prepare graphs.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#prepare-graphs)
    9. [Finalize graphs.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#finalize-graphs)
    10. [Save context into a binary.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#save-context-into-a-binary)
    11. [Load context from a cached binary.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#load-context-from-a-cached-binary)
    12. [Run graphs.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#execute-graphs)
    13. [Free context.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#free-context)
    14. [Terminate backend.](https://docs.qualcomm.com/nav/home/sample_app.html?product=1601111740009302#terminate-backend)

### Load prerequisite shared libraries

QNN SDK provides various shared libraries to access backends and
applications have to load them as needed to run a network.

Create a network in QNN in one of the following ways.

- Build the network directly in your application using QNN APIs.
- Use QNN converters to produce a shared library of a QNN network.

`qnn-sample-app` uses the shared library option. This network can
be produced using one of the QNN converters available in the SDK, and
compiled into a shared library using `qnn-model-lib-generator`.

Note

For Windows users, replace all `.so` files with the analogous
`.dll` file in the following instructions. For more details, see
platform differences.

#### Loading a backend

Shared libraries for various backends including CPU, GPU, HTP, and DSP
are available in the QNN SDK. Every backend that implements QNN APIs
exposes all necessary symbols that can be accessed using dynamic loading
mechanism.

Consider a sample backend shared library
named *libQnnSampleBackend.so*, which can be dynamically loaded as shown
below:

void* libBackendHandle = pal::dynamicloading::dlOpen(
      "libQnnSampleBackend.so", pal::dynamicloading::DL_NOW | pal::dynamicloading::DL_LOCAL);
    
    if (nullptr == libBackendHandle) {
      QNN_ERROR("Unable to load backend. pal::dynamicloading::dlError(): %s",
                pal::dynamicloading::dlError());
      return StatusCode::FAIL_LOAD_BACKEND;
    Copy to clipboard

To load a model as a shared library, let’s consider a sample model
shared library named *libQnnSampleModel.so*, which can be dynamically
loaded as shown below:

void* libModelHandle = pal::dynamicloading::dlOpen(
        "libQnnSampleModel.so", pal::dynamicloading::DL_NOW | pal::dynamicloading::DL_LOCAL);
    
    if (nullptr == libModelHandle) {
      QNN_ERROR("Unable to load model. pal::dynamicloading::dlError(): %s",
                pal::dynamicloading::dlError());
      return StatusCode::FAIL_LOAD_MODEL;
    }
    Copy to clipboard

Optionally, to create a context from a cached binary and execute graphs,
applications can make use of QnnSystem API to retrieve metadata
associated with the context. QnnSystem API can be accessed by loading
the *libQnnSystem.so* shared library as shown below:

void* systemLibraryHandle = pal::dynamicloading::dlOpen(
      "libQnnSystem.so", pal::dynamicloading::DL_NOW | pal::dynamicloading::DL_LOCAL);
    
    if (nullptr == systemLibraryHandle) {
      QNN_ERROR("Unable to load system library. pal::dynamicloading::dlError(): %s",
                pal::dynamicloading::dlError());
      return StatusCode::FAIL_LOAD_SYSTEM_LIB;
    }
    Copy to clipboard

#### Resolving symbols in shared libraries

After the shared libraries are successfully loaded, we can proceed to
resolve all necessary symbols to access QNN APIs.

The below code snippet shows a template to resolve a symbol in a shared
library:

// A generic function to resolve symbols in a library
    template <class T>
    static inline T resolveSymbol(void* libHandle, const char* symName) {
    T ptr = (T)pal::dynamicloading::dlSym(libHandle, symName);
    if (ptr == nullptr) {
       QNN_ERROR("Unable to access symbol [%s]. pal::dynamicloading::dlError(): %s", symName, pal::dynamicloading::dlError());
    }
    return ptr;
    }
    
    // Template for resolving a function of type SampleFnHandleType_t
    typedef ReturnType_t (*SampleFnHandleType_t)(FunctionParameterTypes_t ...);
    SampleFnHandleType_t sampleFn = nullptr;
    sampleFnHandle = resolveSymbol<SampleFnHandleType_t>(libBackendHandle, "QnnSample_API");
    if (nullptr == sampleFnHandle) {
    // Error code indicating failure in symbol resolution
    return StatusCode::FAIL_SYM_FUNCTION;
    }
    Copy to clipboard

The below code snippet shows an example of how to resolve an actual QNN
API:

/* Resolve the symbol for Qnn_ErrorHandle_t QnnInterface_getProviders(const QnnInterface_t*** providerList,
                                                                         uint32_t* numProviders)
       API */
    
    typedef Qnn_ErrorHandle_t (*QnnInterfaceGetProvidersFn_t)(const QnnInterface_t*** providerList,
                                                             uint32_t* numProviders);
    
    QnnInterfaceGetProvidersFn_t getInterfaceProviders {nullptr};
    
    getInterfaceProviders =
    resolveSymbol<QnnInterfaceGetProvidersFn_t>(libBackendHandle, "QnnInterface_getProviders");
    if (nullptr == getInterfaceProviders) {
    return StatusCode::FAIL_SYM_FUNCTION;
    }
    Copy to clipboard

In *qnn-sample-app* source code, all necessary symbols are resolved and
stored in a struct of type QnnFunctionPointers shown below:

typedef struct QnnFunctionPointers {
      // APIs from model output from converters
      // QnnModel_composeGraphs
      ComposeGraphsFnHandleType_t composeGraphsFnHandle;
      // QnnModel_freeGraphsInfo
      FreeGraphInfoFnHandleType_t freeGraphInfoFnHandle;
      // QNN Interface function table containing pointers to all necessary QNN APIs
      // in a backend
      QNN_INTERFACE_VER_TYPE qnnInterface;
      // QNN System Interface function table containing pointers to all QNN System APIs
      QNN_SYSTEM_INTERFACE_VER_TYPE qnnSystemInterface;
    } QnnFunctionPointers;
    Copy to clipboard

The above structure can be found
in ${QNN\_SDK\_ROOT}/examples/QNN/SampleApp/SampleApp/src/SampleApp.hpp.
The rest of the tutorial will assume a variable
named *m\_qnnFunctionPointers* of type *QnnFunctionPointers* that
contains valid function pointers.

### Usage of QNN APIs

This section demonstrates the usage of QNN APIs in a client application.

#### Use QNN Interface to obtain function pointers

QNN Interface mechanism can be used to set up a table of function
pointers to QNN APIs in the backend instead of manually resolving
symbols to each and every API, which makes resolving symbols easy. QNN
Interface can be used as below:

QnnInterface_t** interfaceProviders{nullptr};
    uint32_t numProviders{0};
    // Query for al available interfaces
    if (QNN_SUCCESS !=
       getInterfaceProviders((const QnnInterface_t***)&interfaceProviders, &numProviders)) {
    QNN_ERROR("Failed to get interface providers.");
    return StatusCode::FAIL_GET_INTERFACE_PROVIDERS;
    }
    // Check for validity of returned interfaces
    if (nullptr == interfaceProviders) {
    QNN_ERROR("Failed to get interface providers: null interface providers received.");
    return StatusCode::FAIL_GET_INTERFACE_PROVIDERS;
    }
    if (0 == numProviders) {
    QNN_ERROR("Failed to get interface providers: 0 interface providers.");
    return StatusCode::FAIL_GET_INTERFACE_PROVIDERS;
    }
    bool foundValidInterface{false};
    // Loop through all available interface providers and pick the one that suits the current API
    // version
    for (size_t pIdx = 0; pIdx < numProviders; pIdx++) {
    if (QNN_API_VERSION_MAJOR == interfaceProviders[pIdx]->apiVersion.coreApiVersion.major &&
          QNN_API_VERSION_MINOR <= interfaceProviders[pIdx]->apiVersion.coreApiVersion.minor) {
       foundValidInterface                 = true;
       m_qnnFunctionPointers.qnnInterface = interfaceProviders[pIdx]->QNN_INTERFACE_VER_NAME;
       break;
    }
    }
    if (!foundValidInterface) {
    QNN_ERROR("Unable to find a valid interface.");
    libBackendHandle = nullptr;
    return StatusCode::FAIL_GET_INTERFACE_PROVIDERS;
    }
    Copy to clipboard

QNN System Interface can be used to resolve all symbols related to QNN
System APIs as shown below:

typedef Qnn_ErrorHandle_t (*QnnSystemInterfaceGetProvidersFn_t)(
       const QnnSystemInterface_t*** providerList, uint32_t* numProviders);
    
    QnnSystemInterfaceGetProvidersFn_t getSystemInterfaceProviders{nullptr};
    getSystemInterfaceProviders = resolveSymbol<QnnSystemInterfaceGetProvidersFn_t>(
       systemLibraryHandle, "QnnSystemInterface_getProviders");
    if (nullptr == getSystemInterfaceProviders) {
       return StatusCode::FAIL_SYM_FUNCTION;
    }
    QnnSystemInterface_t** systemInterfaceProviders{nullptr};
    uint32_t numProviders{0};
    if (QNN_SUCCESS != getSystemInterfaceProviders(
                         (const QnnSystemInterface_t***)&systemInterfaceProviders, &numProviders)) {
       QNN_ERROR("Failed to get system interface providers.");
       return StatusCode::FAIL_GET_INTERFACE_PROVIDERS;
    }
    if (nullptr == systemInterfaceProviders) {
       QNN_ERROR("Failed to get system interface providers: null interface providers received.");
       return StatusCode::FAIL_GET_INTERFACE_PROVIDERS;
    }
    if (0 == numProviders) {
       QNN_ERROR("Failed to get interface providers: 0 interface providers.");
       return StatusCode::FAIL_GET_INTERFACE_PROVIDERS;
    }
    bool foundValidSystemInterface{false};
    for (size_t pIdx = 0; pIdx < numProviders; pIdx++) {
       if (QNN_SYSTEM_API_VERSION_MAJOR == systemInterfaceProviders[pIdx]->systemApiVersion.major &&
          QNN_SYSTEM_API_VERSION_MINOR <= systemInterfaceProviders[pIdx]->systemApiVersion.minor) {
       foundValidSystemInterface = true;
       m_qnnFunctionPointers->qnnSystemInterface =
          systemInterfaceProviders[pIdx]->QNN_SYSTEM_INTERFACE_VER_NAME;
       break;
       }
    }
    Copy to clipboard

#### Set up logging

Logging can be set up before a backed is initialized and after a backend
shared library has been dynamically loaded.

To initialize logging, a callback of type *QnnLog\_Callback\_t* has to be
defined. An example is defined below:

void logStdoutCallback(const char* fmt,
                            QnnLog_Level_t level,
                            uint64_t timestamp,
                            va_list argp) {
       const char* levelStr = "";
       switch (level) {
       case QNN_LOG_LEVEL_ERROR:
       levelStr = " ERROR ";
       break;
       case QNN_LOG_LEVEL_WARN:
       levelStr = "WARNING";
       break;
       case QNN_LOG_LEVEL_INFO:
       levelStr = "  INFO ";
       break;
       case QNN_LOG_LEVEL_DEBUG:
       levelStr = " DEBUG ";
       break;
       case QNN_LOG_LEVEL_VERBOSE:
       levelStr = "VERBOSE";
       break;
       case QNN_LOG_LEVEL_MAX:
       levelStr = "UNKNOWN";
       break;
       }
       fprintf(stdout, "%8.1fms [%-7s] ", ms, levelStr);
       vfprintf(stdout, fmt, argp);
       fprintf(stdout, "\n");
    }
    Copy to clipboard

The above callback can be registered with the backend along with a
maximum log level. Sample code to initialize with a max log level of
QNN\_LOG\_LEVEL\_INFO:

Qnn_LogHandle_t logHandle;
    if (QNN_SUCCESS !=
          m_qnnFunctionPointers.qnnInterface.logCreate(logStdoutCallback, QNN_LOG_LEVEL_INFO, &logHandle)) {
    QNN_ERROR("Unable to initialize logging in the backend.");
    return StatusCode::FAILURE;
    }
    Copy to clipboard

#### Initialize backend

Once logging has been successfully initialized, backend can be
initialized as shown below:

1 Qnn_BackendHandle_t backendHandle;
    2 const QnnBackend_Config_t* backendConfigs;
    3 /* Set up any necessary backend configurations */
    4 if (QNN_BACKEND_NO_ERROR != m_qnnFunctionPointers.qnnInterface.backendCreate(logHandle,
    5                                                                              &backendConfigs,
    6                                                                              &backendHandle)) {
    7   QNN_ERROR("Could not initialize backend");
    8   return StatusCode::FAILURE;
    9 }
    Copy to clipboard

#### Initialize Profiling

If profiling is desired, after the backend is initialized, a profile
handle can be set up. This profile handle can be used at a later point
in any API that supports profiling.

A profile handle can be created in the backend with basic profiling
level as shown below:

- ::
    - 1 Qnn\_ProfileHandle\_t profileHandle;
2 if (QNN\_PROFILE\_NO\_ERROR != m\_qnnFunctionPointers.qnnInterface.profileCreate(
3                                   backendHandle, QNN\_PROFILE\_LEVEL\_BASIC, &profileHandle)) {
4   QNN\_WARN(“Unable to create profile handle in the backend.”);
5   return StatusCode::FAILURE;
6 }

#### Create device

Device can be created as shown below:

1 Qnn_DeviceHandle_t deviceHandle {nullptr};
    2 const QnnDevice_Config_t* devConfigArray[] = {&devConfig, nullptr};
    3 Qnn_ErrorHandle_t ret = m_qnnFunctionPointers.qnnInterface.deviceCreate(logHandle, devConfigArray, &deviceHandle);
    4 if (QNN_SUCCESS != ret) {
    5   QNN_ERROR("Failed to create device: %u", qnnStatus);
    6   return StatusCode::FAILURE;
    7 }
    Copy to clipboard

Set devConfig as defined here in QNN HTP Backend API

#### Register op packages

Op packages are way to supply libraries containing ops to backends. They
can be registered as shown below:

1 uint32_t opPackageCount;
    2 char* opPackagePath[opPackageCount];
    3 char* opPackageInterfaceProvider[opPackageCount];
    4 /* Set up required op package paths and interface providers as necessary */
    5 for(uint32_t idx = 0; idx < opPackageCount; idx++) {
    6   if (QNN_BACKEND_NO_ERROR !=
    7         m_qnnFunctionPointers.qnnInterface.backendRegisterOpPackage(backendHandle,
    8                                                                     opPackagePath[idx],
    9                                                                     opPackageInterfaceProvider[idx])) {
    10     QNN_ERROR("Could not register Op Package: %s and interface provider: %s",
    11             opPackagePath[idx],
    12             opPackageInterfaceProvider[idx]);
    13     return StatusCode::FAILURE;
    14   }
    15 }
    Copy to clipboard

#### Create context

A context can be created in a backend as shown below:

1 Qnn_ContextHandle_t context;
    2 Qnn_DeviceHandle_t deviceHandle {nullptr};
    3 const QnnContext_Config_t* contextConfigs;
    4 /* Set up any context configs that are necessary */
    5 if (QNN_CONTEXT_NO_ERROR !=
    6       m_qnnFunctionPointers.qnnInterface.contextCreate(backendHandle,
    7                                                        deviceHandle,
    8                                                        &contextConfigs,
    9                                                        &context)) {
    10   QNN_ERROR("Could not create context");
    11   return StatusCode::FAILURE;
    12 }
    Copy to clipboard

#### Prepare graphs

*qnn-sample-app* relies on the output from one of the converters to
create a QNN network in the backend. *composeGraphsFnHandle* is mapped
to *QnnModel\_composeGraphs* API in the model shared library, which
takes *qnn\_wrapper\_api::GraphInfo\_t\**\*\* as one of the parameters. The
function *composeGraphsFnHandle* will make necessary calls to the
backend to create a network(s). It also writes all necessary
information, like information about input and output tensors related to
the graph, required to execute a graph into the
structure *graphsInfo* as shown in the following code block:

1 /* Structure to retrieve information about graphs, like graph name,
    2    details about input and output tensors preset in libQnnSampleModel.so */
    3 qnn_wrapper_api::GraphInfo_t** graphsInfo;
    4 // No. of graphs present in libQnnSampleModel.so
    5 uint32_t graphsCount;
    6 // true to enable intermediate outputs, false for network outputs only
    7 bool debug;
    8 if (qnn_wrapper_api::ModelError_t::MODEL_NO_ERROR !=
    9         m_qnnFunctionPointers.composeGraphsFnHandle(backendHandle,
    10                                                     m_qnnFunctionPointers.qnnInterface,
    11                                                     context,
    12                                                     &graphsInfo,
    13                                                     &graphsCount,
    14                                                     debug)) {
    15   QNN_ERROR("Failed in composeGraphs()");
    16   return StatusCode::FAILURE;
    17 }
    Copy to clipboard

At this point, the context will contain all the graphs that were present
in *libQnnSampleModel.so*.

#### Finalize Graphs

Graphs that were added in the previous step can be finalized as shown
below:

1 // information about graphs obtained in the previous step
    2 qnn_wrapper_api::GraphInfo_t** graphsInfo;
    3 // No. of graphs obtained in the previous step
    4 uint32_t graphsCount;
    5 /* A valid profile handle if profiling is desired,
    6    nullptr if profiling is not needed */
    7 Qnn_ProfileHandle_t profileHandle;
    8
    9 for (size_t graphIdx = 0; graphIdx < m_graphsCount; graphIdx++) {
    10   if (QNN_GRAPH_NO_ERROR !=
    11     m_qnnFunctionPointers.qnnInterface.graphFinalize(
    12         (*graphsInfo)[graphIdx].graph, profileBackendHandle, nullptr)) {
    13     return StatusCode::FAILURE;
    14   }
    15   /* Extract profiling information if desired and if a valid handle was supplied to finalize
    16      graphs API */
    17 }
    Copy to clipboard

#### Save context into a binary

After all the graphs in a context are finalized, the user application
may choose to save the context into a binary for future use. The
advantage of saving a context is that it can be retrieved in the future
for execution of graphs contained within it without having to finalize
them again. This will save considerable time for initialization during
execution of a network.

The context can be saved as shown below:

1 // Get the expected size of the buffer from the backend in which the context can be saved
    2 if (QNN_CONTEXT_NO_ERROR !=
    3   m_qnnFunctionPointers.qnnInterface.contextGetBinarySize(context, &requiredBufferSize)) {
    4   QNN_ERROR("Could not get the required binary size.");
    5   return StatusCode::FAILURE;
    6 }
    7
    8 // Allocate a buffer of the required size
    9 saveBuffer = (uint8_t*)malloc(requiredBufferSize * sizeof(uint8_t));
    10 if (nullptr == saveBuffer) {
    11   QNN_ERROR("Could not allocate buffer to save binary.");
    12   return StatusCode::FAILURE;
    13 }
    14
    15 auto status = StatusCode::SUCCESS;
    16 uint32_t writtenBufferSize{0};
    17 // Pass the allocated buffer and obtain a copy of the context binary written into the buffer
    18 if (QNN_CONTEXT_NO_ERROR !=
    19   m_qnnFunctionPointers.qnnInterface.contextGetBinary(context,
    20                                                       reinterpret_cast<void*>(saveBuffer),
    21                                                       requiredBufferSize,
    22                                                       &writtenBufferSize)) {
    23  QNN_ERROR("Could not get binary.");
    24  status = StatusCode::FAILURE;
    25 }
    26
    27 // Check if the supplied buffer size is at least as big as the amount of data witten by the backend
    28 if (requiredBufferSize < writtenBufferSize) {
    29   QNN_ERROR(
    30     "Illegal written buffer size [%d] bytes. Cannot exceed allocated memory of [%d] bytes",
    31     writtenBufferSize,
    32     requiredBufferSize);
    33   status = StatusCode::FAILURE;
    34 }
    35
    36 // Use caching utility to save metadata along with the binary buffer from the backend
    37 if (status == StatusCode::SUCCESS &&
    38   tools::datautil::StatusCode::SUCCESS != tools::datautil::writeBinaryToFile(outputPath,
    39                                                                              saveBinaryName + ".bin",
    40                                                                              (uint8_t*)saveBuffer,
    41                                                                              writtenBufferSize)) {
    42   QNN_ERROR("Could not serialize to file.");
    43   status = StatusCode::FAILURE;
    44 }
    Copy to clipboard

#### Load context from a cached binary

A context that was saved into a binary, like in the previous step, can
be loaded as an alternative to creating a new context every time. The
code snippet below demonstrates this step:

1 auto returnStatus   = StatusCode::SUCCESS;
    2 std::shared_ptr<uint8_t> buffer{nullptr};
    3 uint32_t graphsCount {0};
    4 buffer = std::shared_ptr<uint8_t>(new uint8_t[bufferSize], std::default_delete<uint8_t[]>());
    5 if (!buffer) {
    6     QNN_ERROR("Failed to allocate memory.");
    7     return StatusCode::FAILURE;
    8 }
    9
    10 if (tools::datautil::StatusCode::SUCCESS !=
    11     tools::datautil::readBinaryFromFile(
    12     cachedBinaryPath, reinterpret_cast<uint8_t*>(buffer.get()), bufferSize)
    13     QNN_ERROR("Failed to read binary file.");
    14     returnStatus = StatusCode::FAILURE;
    15 }
    16
    17 /* Create a QnnSystemContext handle to access system context APIs. */
    18 QnnSystemContext_Handle_t sysCtxHandle{nullptr};
    19 if (QNN_SUCCESS != m_qnnFunctionPointers.qnnSystemInterface.systemContextCreate(&sysCtxHandle)) {
    20   QNN_ERROR("Could not create system handle.");
    21   returnStatus = StatusCode::FAILURE;
    22 }
    23
    24 /* Retrieve metadata from the context binary through QNN System Context API. */
    25 QnnSystemContext_BinaryInfo_t* binaryInfo{nullptr};
    26 uint32_t binaryInfoSize{0};
    27 if (StatusCode::SUCCESS == returnStatus &&
    28     QNN_SUCCESS != m_qnnFunctionPointers.qnnSystemInterface.systemContextGetBinaryInfo(
    29                      sysCtxHandle,
    30                      static_cast<void*>(buffer.get()),
    31                      bufferSize,
    32                      &binaryInfo,
    33                      &binaryInfoSize)) {
    34     QNN_ERROR("Failed to get context binary info");
    35     returnStatus = StatusCode::FAILURE;
    36 }
    37
    38 qnn_wrapper_api::GraphInfo_t** graphsInfo;
    39 /* Make a copy of the metadata. */
    40 if (StatusCode::SUCCESS == returnStatus &&
    41     !copyMetadataToGraphsInfo(binaryInfo, graphsInfo, graphsCount)) {
    42   QNN_ERROR("Failed to copy metadata.");
    43   returnStatus = StatusCode::FAILURE;
    44 }
    45
    46 /* Release resources associated with previously created QnnSystemContext handle. */
    47 m_qnnFunctionPointers.qnnSystemInterface.systemContextFree(sysCtxHandle);
    48 sysCtxHandle = nullptr;
    49
    50 /* readBuffer contains the binary data that was previously obtained from a backend. Pass this
    51    cached binary data to the backend to recreate the same context. */
    52 if (StatusCode::SUCCESS == returnStatus &&
    53     m_qnnFunctionPointers.qnnInterface.contextCreateFromBinary(backendHandle,
    54                                                                deviceHandle,
    55                                                                (const QnnContext_Config_t**)&contextConfig,
    56                                                                reinterpret_cast<void*>(readBuffer),
    57                                                                bufferSize,
    58                                                                &context,
    59                                                                profileBackendHandle)) {
    60   QNN_ERROR("Could not create context from binary.");
    61   returnStatus = StatusCode::FAILURE;
    62 }
    63
    64 // Optionally, extract profiling numbers if desired
    65 if (ProfilingLevel::OFF != m_profilingLevel) {
    66   extractBackendProfilingInfo(profileBackendHandle);
    67 }
    68
    69 /* Obtain and save graph handles for each graph present in the context based on the saved graph
    70    names in the metadata */
    71 if (StatusCode::SUCCESS == returnStatus) {
    72   for (size_t graphIdx = 0; graphIdx < m_graphsCount; graphIdx++) {
    73     if (QNN_SUCCESS !=
    74         m_qnnFunctionPointers.qnnInterface.graphRetrieve(
    75             context, (*graphsInfo)[graphIdx].graphName, &((*graphsInfo)[graphIdx].graph))) {
    76       QNN_ERROR("Unable to retrieve graph handle for graph Idx: %d", graphIdx);
    77       returnStatus = StatusCode::FAILURE;
    78     }
    79   }
    80 }
    Copy to clipboard

#### Run graphs

After a context has been created, graphs have been added and finalized,
or alternatively, after a context has been retrieved from a binary, one
or more graphs in the context can be executed.

Running a graph involves:

1. Setting up input and output tensors.
2. Populating input data into input tensors.
3. Calling the execute method in the backend.
4. Obtaining outputs and saving them.

This is demonstrated using the code snippet below:

1 // Select a graph from graphsInfo if there are more than one graph in this context
    2 uint32_t graphIdx;
    3 QNN_DEBUG("Starting execution for graphIdx: %d", graphIdx);
    4 Qnn_Tensor_t* inputs  = nullptr;
    5 Qnn_Tensor_t* outputs = nullptr;
    6 // IOTensor utility is used to set up input and output tensor structures
    7 if (iotensor::StatusCode::SUCCESS !=
    8       ioTensor.setupInputAndOutputTensors(&inputs, &outputs, (*graphsInfo)[graphIdx])) {
    9   QNN_ERROR("Error in setting up Input and output Tensors for graphIdx: %d", graphIdx);
    10   returnStatus = StatusCode::FAILURE;
    11   break;
    12 }
    13
    14 // Grab input raw file paths to read input data
    15 auto inputFileList = inputFileLists[graphIdx];
    16 auto graphInfo     = (*graphsInfo)[graphIdx];
    17 if (!inputFileList.empty()) {
    18   /* *qnn-sample-app* reads data based on the batch size until the whole buffer is filled.
    19      If there isn't sufficient data, it pads the rest with zeroes. */
    20   size_t totalCount = inputFileList[0].size();
    21   while (!inputFileList[0].empty()) {
    22     size_t startIdx = (totalCount - inputFileList[0].size());
    23
    24     // IOTensor utility is used to populate input tensors with input data
    25     if (iotensor::StatusCode::SUCCESS !=
    26           m_ioTensor.populateInputTensors(
    27             graphIdx, inputFileList, inputs, graphInfo, inputDataType)) {
    28       returnStatus = StatusCode::FAILURE;
    29     }
    30
    31     if (StatusCode::SUCCESS == returnStatus) {
    32       // Execute the graph in the backend with optional profile handle
    33       QNN_DEBUG("Successfully populated input tensors for graphIdx: %d", graphIdx);
    34       Qnn_ErrorHandle_t executeStatus = QNN_GRAPH_NO_ERROR;
    35       executeStatus = m_qnnFunctionPointers.qnnInterface.graphExecute(graphInfo.graph,
    36                                                                       inputs,
    37                                                                       graphInfo.numInputTensors,
    38                                                                       outputs,
    39                                                                       graphInfo.numOutputTensors,
    40                                                                       profileBackendHandle,
    41                                                                       nullptr);
    42       if (QNN_GRAPH_NO_ERROR != executeStatus) {
    43         returnStatus = StatusCode::FAILURE;
    44       }
    45       if (StatusCode::SUCCESS == returnStatus) {
    46         QNN_DEBUG("Successfully executed graphIdx: %d ", graphIdx);
    47         // IOTensor utility is used to write output tensors to raw files
    48         if (iotensor::StatusCode::SUCCESS !=
    49               ioTensor.writeOutputTensors(graphIdx,
    50                                           startIdx,
    51                                           graphInfo.graphName,
    52                                           outputs,
    53                                           graphInfo.outputTensors,
    54                                           graphInfo.numOutputTensors,
    55                                           outputDataType,
    56                                           graphsCount,
    57                                           outputPath)) {
    58             returnStatus = StatusCode::FAILURE;
    59           }
    60         }
    61       }
    62       if (StatusCode::SUCCESS != returnStatus) {
    63         QNN_ERROR("Execution of Graph: %d failed!", graphIdx);
    64         break;
    65       }
    66     }
    67   }
    68
    69   // Clean up all the tensors after execution is completed
    70   ioTensor.tearDownInputAndOutputTensors(
    71       inputs, outputs, graphInfo.numInputTensors, graphInfo.numOutputTensors);
    72   inputs  = nullptr;
    73   outputs = nullptr;
    74   if (StatusCode::SUCCESS != returnStatus) {
    75     break;
    76   }
    77 }
    Copy to clipboard

IOTensor is a utility provided with the source code
at ${QNN\_SDK\_ROOT}/examples/QNN/SampleApp/SampleApp/src/Utils/IOTensor.cpp.
It exposes a few methods that help with the execution of a graph, which
were used in the previous code snippet:

1. *setupInputAndOutputTensors* to set up structures related to input
and output tensors.
2. *populateInputTensors* to copy input data into input tensor
structures.
3. *tearDownInputAndOutputTensors* to clean up resources associated with
input and output tensors.

Refer to the IOTensor source code for more details about these APIs.

#### Free context

After all the execution is completed, the context can be freed as shown
below:

1 if (QNN_CONTEXT_NO_ERROR !=
    2       m_qnnFunctionPointers.qnnInterface.contextFree(context, profileBackendHandle)) {
    3   QNN_ERROR("Could not free context");
    4   return StatusCode::FAILURE;
    5 }
    Copy to clipboard

#### Terminate backend

Backend can be terminated as shown below:

1 if (QNN_BACKEND_NO_ERROR != m_qnnFunctionPointers.qnnInterface.backendFree(backendHandle)) {
    2   QNN_ERROR("Could not free backend");
    3   return StatusCode::FAILURE;
    4 }
    Copy to clipboard

## SNPE sample app

For C++ API and sample app execution using SNPE, see the [Qualcomm AI Runtime SDK documentation](https://docs.qualcomm.com/nav/home/usergroup8.html?product=1601111740009302).

Last Published: Apr 02, 2026

[Previous Topic
Add postprocessing support for a custom model](https://docs.qualcomm.com/bundle/publicresource/80-70029-15B/topics/add-postprocessing-support-custom-model.md) [Next Topic
Use AI Hub models with the GStreamer API](https://docs.qualcomm.com/bundle/publicresource/80-70029-15B/topics/use-ai-hub-models-with-gstreamer.md)