# InferenceSet I/O example
The following document describes `AIC100` example named
`InferenceSetIOBuffersExample.cpp`.
This example contains a single C++ file and a `CMakeLists.txt` that
you can use for compiling as part of `Qualcomm Cloud AI 100`
distributed Platform SDK.
InferenceSetIOBuffersExample.cpp
1 //-----------------------------------------------------------------------------
2 // Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.
3 // SPDX-License-Identifier: BSD-3-Clause-Clear
4 //-----------------------------------------------------------------------------
5
6 #include <string>
7 #include <vector>
8 #include <iostream>
9 #include <random>
10 #include "QAicApi.hpp"
11
12 namespace {
13
14 /**
15 * used to generate random data into input buffers
16 * Input buffers are uint8_t arrays
17 */
18 struct RandomGen final {
19 static constexpr const int from = std::numeric_limits<uint8_t>::min();
20 static constexpr int to = std::numeric_limits<uint8_t>::max();
21 std::random_device randdev;
22 std::mt19937 gen;
23 std::uniform_int_distribution<uint8_t> distr;
24 explicit RandomGen() : gen(randdev()), distr(from, to) {}
25 [[nodiscard]] auto next() { return distr(gen); }
26 };
27
28 /**
29 * Simple helper to return true if the buffer mapping instance is an input one
30 * @param bufmap buffer mapping instance
31 * @return true if the instance is an input buffer one.
32 */
33 [[nodiscard]] bool isInputBuffer(const qaic::rt::BufferMapping &bufmap) {
34 return bufmap.ioType == BUFFER_IO_TYPE_INPUT;
35 }
36
37 /**
38 * Helper function to print input/output buffer counts so far, zero based
39 * @param bufmap buffer map instance
40 * @param inputCount input count to use if the instance is input
41 * @param outputCount output count to use if the instance is output
42 * @return string formatted using the above info.
43 */
44 [[nodiscard]] std::string getPrintName(const qaic::rt::BufferMapping &bufmap,
45 const std::size_t inputCount,
46 const std::size_t outputCount) {
47 using namespace std::string_literals;
48 return isInputBuffer(bufmap) ? ("Input "s + std::to_string(inputCount))
49 : ("Output "s + std::to_string(outputCount));
50 }
51
52 /**
53 * Populate input, output vectors with QBuffer information
54 * @param bufmap Buffer mapping instance
55 * @param buf Actual QBuffer that was generated at callsite/caller.
56 * @param inputBuffers Vector to use in case this is input instance
57 * @param outputBuffers Vector to use in case this is an output instance
58 */
59 void populateVector(const qaic::rt::BufferMapping &bufmap, const QBuffer &buf,
60 std::vector<QBuffer> &inputBuffers,
61 std::vector<QBuffer> &outputBuffers) {
62 if (isInputBuffer(bufmap)) {
63 inputBuffers.push_back(buf);
64 } else {
65 outputBuffers.push_back(buf);
66 }
67 }
68
69 /**
70 * Given a buffer and size, populate it with random [0..128] random data
71 * @param buf buffer to populate
72 * @param sz size of this buffer
73 */
74 void populateBufWithRandom(uint8_t *buf, const std::size_t sz) {
75 RandomGen gen;
76 for (auto iter = buf; iter < buf + sz; ++iter) {
77 *iter = gen.next();
78 }
79 }
80
81 /**
82 * Prepare buffers, vectors given a single buffer mapping. Depending on the
83 * input/output instance of the buffer mapping, handle logic accordingly.
84 * Only inputbuffers needs to be populated with random data.
85 * @param bufmap buffer mapping passed as const
86 * @param inputCount input buffers counter
87 * @param outputCount output buffers counter
88 * @param inputBuffers input vector of QBuffer to append to new QBuffer
89 * @param outputBuffers output vector of QBuffer to append to new QBuffer
90 */
91 void prepareBuffers(const qaic::rt::BufferMapping &bufmap,
92 std::size_t &inputCount, std::size_t &outputCount,
93 std::vector<QBuffer> &inputBuffers,
94 std::vector<QBuffer> &outputBuffers) {
95 std::cout << getPrintName(bufmap, inputCount, outputCount) << '\n';
96 std::cout << "\tname = " << bufmap.bufferName << '\n';
97 std::cout << "\tsize = " << bufmap.size << '\n';
98 QBuffer buf{bufmap.size, new uint8_t[bufmap.size]}; // Need to dealloc
99 populateVector(bufmap, buf, inputBuffers, outputBuffers);
100 //
101 // Provide the input to the inference in "inputBuffers". Here random data
102 // is used. For providing input as file, use a different api. This example
103 // is for input in memory.
104 //
105 if (isInputBuffer(bufmap)) {
106 populateBufWithRandom(buf.buf, buf.size);
107 ++inputCount;
108 } else {
109 ++outputCount;
110 }
111 }
112
113 /**
114 * Given input and output buffers, release all heap allocated
115 * @param inputBuffers vector of QBuffers - inputs
116 * @param outputBuffers vector of Qbuffers - outputs
117 */
118 void releaseBuffers(std::vector<QBuffer> &inputBuffers,
119 std::vector<QBuffer> &outputBuffers) {
120 const auto release([](const QBuffer &qbuf) { delete[] qbuf.buf; });
121 std::for_each(inputBuffers.begin(), inputBuffers.end(), release);
122 std::for_each(outputBuffers.begin(), outputBuffers.end(), release);
123 }
124
125 /**
126 * Given buffer mapping instance, return true if this instance doesn't
127 * contain input or output buffers (e.g. it contains uninitialized or invalid)
128 * @param bufmap buffer mapping instance
129 * @return true if the buffer mapping instance doesn't contain a valid buffer
130 */
131 [[nodiscard]] bool notInputOrOutput(const qaic::rt::BufferMapping &bufmap) {
132 const std::initializer_list<QAicBufferIoTypeEnum> bufTypes{
133 BUFFER_IO_TYPE_INPUT, BUFFER_IO_TYPE_OUTPUT};
134 const auto func([type = bufmap.ioType](const auto v) { return v == type; });
135 return std::none_of(bufTypes.begin(), bufTypes.end(), func);
136 }
137
138 } // namespace
139
140 int main([[maybe_unused]] int argc, [[maybe_unused]] char *argv[]) {
141 // *** QID ***
142 QID qid = 0;
142 std::vector<QID> qidList{qid};
144
145 // *** QPC ***
146 constexpr const char *qpcPath =
147 "/opt/qti-aic/test-data/aic100/v2/2nsp/2nsp-conv-hmx";
148 auto qpc = qaic::rt::Qpc::Factory(qpcPath);
149
150 // *** CONTEXT ***
151 constexpr QAicContextProperties_t *NullProp = nullptr;
152 auto context = qaic::rt::Context::Factory(NullProp, qidList);
153
154 // *** INFERENCE SET ***
155 constexpr uint32_t setSize = 10;
156 constexpr uint32_t numActivations = 1;
157 auto inferenceSet = qaic::rt::InferenceSet::Factory(
158 context, qpc, qidList.at(0), setSize, numActivations);
159
160 // *** SETUP IO BUFFERS ***
161 qaic::rt::shInferenceHandle submitHandle;
162 auto status = inferenceSet->getAvailable(submitHandle);
163 if (status != QS_SUCCESS) {
164 std::cerr << "Error obtaining Inference Handle\n";
165 return -1;
166 }
167 std::size_t numInputBuffers = 0;
168 std::size_t numOutputBuffers = 0;
169 std::vector<QBuffer> inputBuffers, outputBuffers;
170 const auto &bufferMappings = qpc->getBufferMappings();
171 for (const auto &bufmap : bufferMappings) {
172 if (notInputOrOutput(bufmap)) {
173 continue;
174 }
175 prepareBuffers(bufmap, numInputBuffers, numOutputBuffers, inputBuffers,
176 outputBuffers);
177 }
178 submitHandle->setInputBuffers(inputBuffers);
179 submitHandle->setOutputBuffers(outputBuffers);
180
181 // *** SUBMISSION ***
182 constexpr uint32_t inferenceId = 0; // also named as request ID
183 status = inferenceSet->submit(submitHandle, inferenceId);
184 std::cout << status << '\n';
185
186 // *** COMPLETION ***
187 qaic::rt::shInferenceHandle completedHandle;
188 status = inferenceSet->getCompletedId(completedHandle, inferenceId);
189 std::cout << status << '\n';
190 status = inferenceSet->putCompleted(std::move(completedHandle));
191 std::cout << status << '\n';
192
193 // *** GET OUTPUT ***
194 //
195 // At this point, the output is available in "outputBuffers" and can be
196 // consumed.
197 //
198
199 // *** Release user allocated buffers ***
200 releaseBuffers(inputBuffers, outputBuffers);
201 return 0;
202 }
## Main flow
The main function has nine parts and uses a few helper functions defined in the top anonymous namespace. The following shows the overall structure of `main()`.
int main([[maybe_unused]] int argc, [[maybe_unused]] char *argv[]) {
// *** QID ***
// *** QPC ***
// *** CONTEXT ***
// *** INFERENCE SET ***
// *** SETUP IO BUFFERS ***
// *** SUBMIT ***
// *** COMPLETION ***
// *** GET OUTPUT ***
// *** Release user allocated buffers ***
return 0;
}
Copy to clipboard
This section describes the nine parts in the main flow. See [Helper functions](https://docs.qualcomm.com/doc/80-99100-3/topic/index_qaic-inference-set-io-buffers.html#helper-functions) for more information about the helper functions.
### QID
The first part of the `main()` example picks `QID 0`. This is
usually the first Enumerated device ID.
Though the API is capable of accepting a list of *QID* ‘s, this
example only passes a single one in the `vector<> int` container.
### QPC
[QPC](https://docs.qualcomm.com/doc/80-99100-3/topic/index_runtime.html#reference-to-qpc) is a container file that includes various
parts of the compiled network.
This example hardcodes the path, but you can change it or pass it to the program by using an environment variable or command-line arguments
`qaic::rt::Qpc::Factory` accepts a path to the QPC and
returns a QPC object to use in the next steps.
### Context
`QAIC` Runtime requires that you create a [Context](https://docs.qualcomm.com/doc/80-99100-3/topic/index_runtime.html#reference-to-context) object
and passed it to the various APIs.
In this phase, `qaic::rt::Context::Factory` obtains a new
instance of `Context`. Pass `NullProp` for no special `QAicContextProperties_t`
attributes, along with the previously instantiated QID vector.
### Inference set
Creating an instance of [InferenceSet](https://docs.qualcomm.com/doc/80-99100-3/topic/index_runtime.html#reference-to-inference-set) is
the next step.
`inferenceSet` is a top-level entity when it comes to running inferences on hardware.
The example sets the size of the software and hardware backlog as 10
possible pending buffers.
It requests a single activation. This means that the provided
program (encapsulated in `qpc`), runs as a single instance
on the hardware, using the single QID provided when creating the
InferenceSet instance.
### Setup I/O buffers
This part of the code sets up the input and output buffers. Your application is responsible for
allocating the buffers and also deallocating the buffers before application tear-down.
This part has the following subsections:
1. Obtain [InferenceHandle](https://docs.qualcomm.com/doc/80-99100-3/topic/index_runtime.html#reference-to-inference-handle) to submit
the I/O buffers once these created.
2. Iterate over each `BufferMapping` instance to obtain information needed to allocate and populate new buffers, storing the result in `inputBuffers` or `outputBuffers`.
3. `prepareBuffers` allocates the input and output buffers, then `submitHandle` submits them for use during inference.
In this example, the helper functions populate the input buffers with random data just to demonstrate the capabilities of the system.
### Submission
This is the part where the actual submission request is happening.
`inferenceSet` submits the request and passes the `submitHandle`
and user defined `inferenceId`.
### Completion
This is a blocking call to wait on inference completion and device
output’s buffers received in the Application.
`inferenceSet` obtains the `completedHandle` by passing the
`inferenceId` from the previous section.
User is responsible to return the `completedHandle` back to the Runtime
pool and doing so by calling `putCompleted` using `inferenceSet`.
### Obtaining output
This example doesn’t process the obtained Output buffers; a real application would consume this output data.
### Release user allocated buffers
Because the buffers are user-allocated buffers using the System’s Heap,
you are also in charge of releasing these buffers as shown in the last phase of this example.
## Helper functions
The InferenceSetIOBuffersExample.cpp example uses the following helper functions and constructs:
- `RandomGen` : Generates random input buffer data.
- `isInputBuffer` : Queries if a specific `BufferMapping` instance is
an input one.
- `getPrintName` : Returns input or output strings for standard output
printing.
- `populateVector` : Populates inputs or output `vector <>` containers.
- `populateBufWithRandom` : Uses `RandomGen` to populate a buffer you specify with random values.
- `prepareBuffers` : Iterates over the `BufferMappings` container, populates the `inputBuffers` and `outputBuffers` for each `BufferMapping` instance, then calls `populateBufWithRandom` to populate the input buffers with random data.
- `releaseBuffers` : Iterates over allocated
inputs/outputs and release/delete buffers (returns memory back to the
system’s heap).
- `notInputOrOutput` : Returns `true` if
a `BufferMapping` instance has an `ioType` true if this instance is neither input nor
output. For example, an uninitialized or invalid type. The caller uses this return value to skip such instances.
## Compile and run commands
Copy the `InferenceSetIOBuffersExample.cpp` and `CMakeLists.txt` in
a folder, then compile the example with following commands:
mkdir build
cd build
cmake ..
make -j 8
Copy to clipboard
Finally, run the executable `./inference-set-io-buffers-example`,
accordingly change the `qpcPath`.
Last Published: Aug 25, 2026
[Previous Topic
Examples](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/index_example.md) [Next Topic
QAicInferenceSet Example](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/index_qaic-inference-set-group.md)