# Block Ops Definitions This page documents detailed operator definitions for block ops supported by QAIRT tools. Block ops use the special domain name **qti\_aisw**. ## Buffer This block op accumulates inputs across inferences into a buffer of size buffer\_size and outputs the buffer with the collected inputs. When the buffer is full the oldest existing inputs in the buffer are removed to make space for the incoming new input. The number of inputs to remove is determined by stride. The remaining inputs are shifted in the buffer to maintain the order they were received. - **Signature** - Buffer(input data: TensorT[batch, height, width, channel], input reset: Boolean, parameter buffer_size: uint32, parameter buffer_dim: uint32, parameter buffer_padding: uint32, parameter stride: uint32, parameter mode: enum(uint32) → output data: TensorT[batch, height, width, channel]) Copy to clipboard - **Inputs** - | Name | Description | | --- | --- | | **inp: T (Tensor)** | Input data to be accumulated.

| | **reset: Boolean** | The reset input determines if the buffer should be reset.


When set to true all frames in the buffer are removed.

| - **Outputs** - | Name | Description | | --- | --- | | **out: T (Tensor)** | Output after inputs are accumulated.

| - **Attributes/Parameters** - | Name | Description | | --- | --- | | **buffer\_size: uint32** | Determines the number of inputs that a buffer can store.

| | **buffer\_dim: uint32** | Determines the dimension that inputs are accumulated on.

| | **buffer\_padding: uint32** | Determines the number of frames to pad with 0’s to the buffer initially or after reset.

| | **stride: uint32** | Determines the number of inputs to remove from the buffer when the buffer is full to make space for the new incoming input.


The oldest existing inputs which reside at the beginning of the buffer are removed.


After removal the remaining inputs are kept in the order they were received.

| | **mode: enum (uint32)** | Determines blocking behavior. How the buffer is populated differs between the modes when the buffer is not full.


When the buffer is full eviction and population behavior is the same for all modes.



When the buffer is full the number of inputs removed is determined by stride.


Eviction behavior is the same for all modes where the oldest existing inputs are removed from the beginning of the
buffer.


Population behavior is also the same for all modes when the buffer is full.


For example, a fully populated buffer with 3 slots (0,1,2) and a stride value of 2 will have the inputs at
slot 0 and slot 1 removed and the input at slot 2 will now be at slot 0.


The incoming input is then placed at slot 1.


The next incoming input will then be placed at slot 2.


When the buffer is full again the same process is repeated.


Note


NON\_BLOCKING\_LEFT and NON\_BLOCKING\_RIGHT are zero filled for the output if the buffer is not completely full.

| - **Type Constraints** - | TypeName | Datatypes | | --- | --- | | **T** | **tensor(float32)** | - **Supported Backends** - - QNN-CPU - QNN-LPAI - QNN-HTP Note It is not feasible to mimic the Buffer block op in ONNX Runtime as it has blocking and stateful behavior. Hence, it is implemented as an identity op in ONNX Runtime and the output of this op may not match the expected output in ONNX Runtime. Note HTP backend only supports NON\_BLOCKING modes of Buffer Op. ## MaskedSoftmax This block op applies a Softmax operation on masked portions of the input tensor. For each batch the mask tensor is broadcast on the input before softmax computation. A mask tensor must be provided in either an UNCOMPRESSED or COMPRESSED format depending on the parameter mode selected. See input mask for details on how a boolean mask can be converted to an UNCOMPRESSED or COMPRESSED mask tensor. - **Signature** - MaskedSoftmax(input data: TensorT[batch, height, width, channel], input mask: TensorT1[batch, M], parameter mode: enum(int) → output data: TensorT[batch, height, width, channel]) Copy to clipboard - **Inputs** - | Name | Description | | --- | --- | | **data: T (Tensor)** | Input data to be masked and softmax to be applied to.
| | **mask: T1 (Tensor)** | The representation of this 2D tensor is determined by the parameter mode selected.


When parameter mode is set to UNCOMPRESSED, then M = channel or when set to COMPRESSED, then
M = number of sequences.
| Consider a boolean mask where a mask value of 1 indicates the dimension on which Softmax should be performed and a mask value of 0 indicates the dimension that Softmax will not be performed. An uncompressed mask can be made from a boolean mask tensor by adding -1 or subtracting by 1 element-wise and multiplying the intermediate result element-wise by a large value. mask = [[1,1,1,0,1]] uncompressed_mask = (mask .+ -1) .* 10000 // uncompressed_mask = [[0,0,0,-10000,0]] Copy to clipboard A compressed mask can be made from multiple boolean mask tensors of vector lengths that are concatenated into a single batch and summed across the 2nd axis. Consider the following: Let there be 3 mask tensors that correspond to sequences of inputs that were used to make in[0] where 0’s represent where padding was added to make them the max sequence length. mask1 = [1,0,0,0] mask2 = [1,1,1,0] mask3 = [1,1,1,1] Copy to clipboard The concatenated mask would then be the following: concatenated_mask = [ [1,0,0,0], [1,1,1,0], [1,1,1,1]] Copy to clipboard The compressed mask representation would be made from summing across the 2nd axis: compressed_mask = [[1,3,4]] Copy to clipboard - **Outputs** - | Name | Description | | --- | --- | | **out: T (Float32 Tensor)** | Output after input is masked and softmax to be applied. | - **Attributes/Parameters** - | Name | Description | | --- | --- | | **mode: enum(int)** | Determines the format of the input mask. See input mask for details on the format of the mask tensor.
| - **Type Constraints** - | TypeName | Datatypes | | --- | --- | | **T** | **tensor(float32)** | | **T1** | **tensor(int32)** | - **Supported Backends** - - QNN-CPU - QNN-AIC ## StatefulGRU This Block op computes an one-layer GRU. **Equations:** \[ \begin{align}\begin{aligned}\begin{split}{\Large z\_t = f(X\_t\*(Wz^T) + H\_{t-1}\*(Rz^{T}) + Wbz + Rbz)} \\ \\\end{split}\\\begin{split}{\Large r\_t = f(X\_t\*(Wr^T) + H\_{t-1}\*(Rr^T) + Wbr + Rbr)} \\ \\\end{split}\\\begin{split}{\Large h\_{t} = g(X\_t\*(Wh^T) + (r\_t \odot H\_{t-1})\*(Rh^T) + Rbh + Wbh) \text{, When linear\_before\_reset = 0}} \\ \\\end{split}\\\begin{split}{\Large h\_{t} = g(X\_t\*(Wh^T) + (r\_t \odot (H\_{t-1}\*(Rh^T) + Rbh)) + Wbh) \text{, When linear\_before\_reset != 0}} \\ \\\end{split}\\\begin{split}{\Large H\_t = (1 - z\_t) \odot h\_t + z\_t \odot H\_{t-1}} \\ \\\end{split}\end{aligned}\end{align} \] Where, - \(X\_{t}\) - input tensor - \(z\_{t}\) - update gate - \(r\_{t}\) - reset gate - \(h\_{t}\) - hidden gate - \(t\) - time step (\(t-1\) means previous time step) - \(f\) - sigmoid activation function - \(g\) - tanh activation function - W[zrh] - W parameter weight matrix for update, reset, and hidden gates - R[zrh] - R recurrence weight matrix for update, reset, and hidden gates - Wb[zrh] - W bias vectors for update, reset, and hidden gates - Rb[zrh] - R bias vectors for update, reset, and hidden gates - WB[zrh] - W parameter weight matrix for backward update, reset, and hidden gates - RB[zrh] - R recurrence weight matrix for backward update, reset, and hidden gates - WBb[zrh] - W bias vectors for backward update, reset, and hidden gates - RBb[zrh] - R bias vectors for backward update, reset, and hidden gates - H - Hidden state - num\_directions - 2 if direction == bidirectional else 1 - \(\odot\) - element-wise product of two vectors. **Activation functions:** - Tanh(x) - \((1 - e^{-2x})/(1 + e^{-2x})\) - Sigmoid(x) - \(1/(1 + e^{-x})\) **References:** ONNX: [ops::GRU](https://onnx.ai/onnx/operators/onnx__GRU.html#gru-7) **Signature** StatefulGru( input X: TensorT[seq_length, batch_size, input_size], input W: TensorT[num_directions, 3*hidden_size, input_size], input R: TensorT[num_directions, 3*hidden_size, hidden_size], parameter hidden_size: int, input B: TensorT[num_directions, 6*hidden_size] = None, input sequence_lens: TensorT1[batch_size] = None, input initial_h: TensorT[num_directions, batch_size, hidden_size] = None, input reset: BOOL = False, parameter clip: float = None, parameter direction: str = "forward", parameter linear_before_reset: int = 0, ) → output Y: TensorT[seq_length, num_directions, batch_size, hidden_size], output Y_h: TensorT[num_directions, batch_size, hidden_size] Copy to clipboard - **Inputs** - | Name | Description | | --- | --- | | **X: T (Tensor)** | The input sequence.
| | **W: T (Tensor)** | The weight tensor for the gates. Concatenation of W[zrh] and WB[zrh] (if bidirectional) along dimension 0.
| | **R: T (Tensor)** | The recurrence weight tensor. Concatenation of R[zrh] and RB[zrh] (if bidirectional) along dimension 0.
| | **B: T (Tensor)** | The bias tensor for input gate. Concatenation of [Wb[zrh], Rb[zrh]] and [WBb[zrh], RBb[zrh]] (if bidirectional) along dimension 0.
| | **sequence\_lens : T1 (Tensor)** | Optional tensor specifying lengths of the sequences in a batch.
| | **initial\_h: T (Tensor)** | Optional initial value of the hidden.
| | **reset: Boolean** | Determines if the internal state should be reset from the beginning of an inference pass.


When set to true the internal state is reset by the input initial\_h if it is provided, otherwise it is set to all zero values.
When set to false the internal state is gotten from last step’s final\_hidden tensor.


This input is used to indicate the reset of the internal state at the beginning of an inference pass across all batch elements at time-step 0.


Note


The reset input is ignored in ONNX Runtime and the behavior is same as Normal GRU. | - **Outputs** - | Name | Description | | --- | --- | | **Y: T (Tensor)** | A tensor that concats all the intermediate output values of the hidden.
| | **Y\_h: T (Tensor)** | The last output value of the hidden.
| - **Attributes/Parameters** - | Name | Description | | --- | --- | | **hidden\_size: int** | Number of neurons in the hidden layer
| | **clip: float** | Cell clip threshold. Clipping bounds the elements of a tensor in the range of [-threshold, +threshold] and is applied to the input of activations. No clip if not specified.
| | **direction: str** | Specify if the RNN is forward, reverse, or bidirectional. Must be one of forward (default), reverse, or bidirectional.
| | **linear\_before\_reset: int** | When computing the output of the hidden gate, apply the linear transformation before multiplying by the output of the reset gate.
| - **Type Constraints** - | TypeName | Datatypes | | --- | --- | | **T** | **tensor(float32)** | | **T1** | **tensor(int32)** | - **Supported Backends** - - QNN-CPU - QNN-LPAI - QNN-HTP ## StatefulLSTM This Block op computes an one-layer LSTM. **Equations:** \[ \begin{align}\begin{aligned}\begin{split}{\Large i\_t = f(X\_t\*(Wi^T) + H\_{t-1}\*(Ri^T) + Pi \odot C\_{t-1} + Wbi + Rbi)} \\ \\\end{split}\\\begin{split}{\Large f\_t = f(X\_t\*(Wf^T) + H\_{t-1}\*(Rf^T) + Pf \odot C\_{t-1} + Wbf + Rbf)} \\ \\\end{split}\\\begin{split}{\Large c\_t = g(X\_t\*(Wc^T) + H\_{t-1}\*(Rc^T) + Wbc + Rbc)} \\ \\\end{split}\\\begin{split}{\Large C\_t = f\_t \odot C\_{t-1} + i\_t \odot c\_t} \\ \\\end{split}\\\begin{split}{\Large o\_t = f(X\_t\*(Wo^T) + H\_{t-1}\*(Ro^T) + Po \odot C\_t + Wbo + Rbo)} \\ \\\end{split}\\\begin{split}{\Large H\_t = o\_t \odot h(C\_t)} \\ \\\end{split}\end{aligned}\end{align} \] Where, - \(X\_{t}\) - input tensor - \(i\_{t}\) - input gate - \(o\_{t}\) - output gate - \(f\_{t}\) - forget gate - \(c\_{t}\) - cell gate - \(t\) - time step (\(t-1\) means previous time step) - \(f\) - sigmoid activation function - \(g\) - tanh activation function - \(h\) - tanh activation function - W[iofc] - W parameter weight matrix for input, output, forget, and cell gates - R[iofc] - R recurrence weight matrix for input, output, forget, and cell gates - Wb[iofc] - W bias vectors for input, output, forget, and cell gates - Rb[iofc] - R bias vectors for input, output, forget, and cell gates - P[iof] - P peephole weight vector for input, output, and forget gates - WB[iofc] - W parameter weight matrix for backward input, output, forget, and cell gates - RB[iofc] - R recurrence weight matrix for backward input, output, forget, and cell gates - WBb[iofc] - W bias vectors for backward input, output, forget, and cell gates - RBb[iofc] - R bias vectors for backward input, output, forget, and cell gates - PB[iof] - P peephole weight vector for backward input, output, and forget gates - H - Hidden state - num\_directions - 2 if direction == bidirectional else 1 - \(\odot\) - element-wise product of two vectors. **Activation functions:** - Tanh(x) - \((1 - e^{-2x})/(1 + e^{-2x})\) - Sigmoid(x) - \(1/(1 + e^{-x})\) **References:** ONNX: [ops::LSTM](https://onnx.ai/onnx/operators/onnx__LSTM.html#lstm-7) - **Signature** - StatefulLstm( input X: TensorT[seq_length, batch_size, input_size], input W: TensorT[num_directions, 4*hidden_size, input_size], input R: TensorT[num_directions, 4*hidden_size, hidden_size], parameter hidden_size: int, input B: TensorT[num_directions, 8*hidden_size] = None, input sequence_lens: TensorT1[batch_size] = None, input initial_h: TensorT[num_directions, batch_size, hidden_size] = None, input initial_c: TensorT[num_directions, batch_size, hidden_size] = None, input P: TensorT[num_directions, 3*hidden_size] = None, input reset: BOOL = False, parameter clip: float = None, parameter direction: str = "forward", parameter input_forget: int = 0, ) → output Y: TensorT[seq_length, num_directions, batch_size,hidden_size], output Y_h: TensorT[num_directions, batch_size, hidden_size], output Y_c: TensorT[num_directions, batch_size, hidden_size] Copy to clipboard - **Inputs** - | Name | Description | | --- | --- | | **X: T (Tensor)** | The input sequence.
| | **W: T (Tensor)** | The weight tensor for the gates. Concatenation of W[iofc] and WB[iofc] (if bidirectional) along dimension 0.
| | **R: T (Tensor)** | The recurrence weight tensor. Concatenation of R[iofc] and RB[iofc] (if bidirectional) along dimension 0.
| | **B: T (Tensor)** | The bias tensor for input gate. Concatenation of [Wb[iofc], Rb[iofc]], and [WBb[iofc], RBb[iofc]] (if bidirectional) along dimension 0.
| | **sequence\_lens : T1 (Tensor)** | Optional tensor specifying lengths of the sequences in a batch.
| | **initial\_h: T (Tensor)** | Optional initial value of the hidden.
| | **initial\_c: T (Tensor)** | Optional initial value of the cell.
| | **P: T (Tensor)** | The weight tensor for peepholes. Concatenation of P[iof] and PB[iof] (if bidirectional) along dimension 0.
| | **reset: Boolean** | Determines if the internal cell and hidden state should be reset from the beginning of an inference pass.


When set to true the internal states are reset by the input initial\_h and initial\_c if they are provided, otherwise they are set to all zero values.
When set to false the internal states are gotten from last step’s final\_hidden and final\_cell tensors.


This input is used to indicate the reset of the internal states at the beginning of an inference pass across all batch elements at time-step 0.


Note


The reset input is ignored in ONNX Runtime and the behavior is same as Normal LSTM. | - **Outputs** - | Name | Description | | --- | --- | | **Y: T (Tensor)** | A tensor that concats all the intermediate output values of the hidden.
| | **Y\_h: T (Tensor)** | The last output value of the hidden.
| | **Y\_c: T (Tensor)** | The last output value of the cell.
| - **Attributes/Parameters** - | Name | Description | | --- | --- | | **hidden\_size: int** | Number of neurons in the hidden layer
| | **clip: float** | Cell clip threshold. Clipping bounds the elements of a tensor in the range of [-threshold, +threshold] and is applied to the input of activations. No clip if not specified.
| | **direction: str** | Specify if the RNN is forward, reverse, or bidirectional. Must be one of forward (default), reverse, or bidirectional.
| | **input\_forget: int** | Couple the input and forget gates if 1.
| - **Type Constraints** - | TypeName | Datatypes | | --- | --- | | **T** | **tensor(float32)** | | **T1** | **tensor(int32)** | - **Supported Backends** - - QNN-CPU - QNN-LPAI - QNN-HTP Last Published: Oct 10, 2025 [Previous Topic StatefulLstm Usage](https://docs.qualcomm.com/bundle/publicresource/80-63442-50/topics/blockop_onnx_usage.md) [Next Topic Backend supplements](https://docs.qualcomm.com/bundle/publicresource/80-63442-50/topics/operations.md)