# Qualcomm AI Engine Direct Model Execution on Windows on Snapdragon
Source: [https://docs.qualcomm.com/doc/80-64748-1/topic/model_execution_windows.html](https://docs.qualcomm.com/doc/80-64748-1/topic/model_execution_windows.html)
qnn_model_execution_on_windows
# Qualcomm AI Engine Direct Stable Diffusion Model Execution on Windows on Snapdragon
Download
This guide will provide details on how to execute the Stable Diffusion model on a Windows on Snapdragon device uses the Qualcomm AI Engine Direct SDK.
Please note that for the rest of this document the term Qualcomm Neural Network (QNN) will be interchangeably used with Qualcomm AI Engine Direct SDK.
# Prerequisites
1. QNN SDK
2. QNN context binary generated for the three models that make up Stable Diffusion: text encoder, U-Net, and variational autoencoder decoder
# Platform requirements
- Surface Pro9
- OS build: 22621.169
- Windows Feature Experience Pack : 1000.22632.1000.0
- Windows On Snapdragon : SC8280X
- qcadsprpc file version : should be 1.0.3530.9800 or greater
### To get qcadsprpc version, follow the below steps:
1. Update Windows OS and Windows drivers with the latest versions.
2. Open the **System Information** menu and select **Software Environment** > **System Drivers**.
3. Locate the driver named “qcadsprpc” and note the file path.
4. Open Windows Explorer and navigate to the file path noted in Step 3.
5. Left-click the qcadsprpc8280.sys file and hover over it to see the file version number (alternatively, you can right-click the file > Select **Properties** > Select **Details** and view the file version).
6. Confirm that the file version is 1.0.3530.9800 or above.
# Workflow
1. (Optional) Set up a Jupyter notebook. The following code has been validated on a jupyter notebook on Windows environment. Hence, we recommend setting up a Jupyter notebook following these instructions.
2. Run models in a Stable Diffusion pipeline. Given a user prompt, execute the models as a Stable Diffusion pipeline on QNN HTP on the WoS platform to produce an image.
## Set up Jupyter notebook
1. Download Visual Studio Build Tools (2022): [https://visualstudio.microsoft.com/visual-cpp-build-tools/](https://visualstudio.microsoft.com/visual-cpp-build-tools/)
2. During installation, ensure that **Desktop development with C++** is selected in the Visual Studio Installer.
3. `pip install notebook`
4. `jupyter notebook`
## Preparing all the binaries and libraries for execution
In [ ]:
# Copy the required libraries and binaries to libs folder
import shutil
import os
execution_ws = os.getcwd()
SDK_dir = execution_ws + "qnn_assets\\" + #""
lib_dir = SDK_dir + "\\lib\\aarch64-windows-msvc\\"
binary = SDK_dir + "\\bin\\aarch64-windows-msvc\qnn-net-run.exe"
skel = SDK_dir + "\\lib\\hexagon-v68\\unsigned\libQnnHtpV68Skel.so"
des_dir = execution_ws + "qnn_assets\\QNN_binaries"
# Copy necessary libraries to a common location
libs = ["QnnHtp.dll", "QnnHtpNetRunExtensions.dll", "QnnHtpPrepare.dll", "QnnHtpV68Stub.dll"]
for lib in libs:
shutil.copy(lib_dir+lib, des_dir)
# Copy binary
shutil.copy(binary, des_dir)
# Copy Skel
shutil.copy(skel, des_dir)
Copy to clipboard
# Execute models in Stable Diffusion pipeline using Qualcomm AI Engine Direct
This section continues using the Qualcomm AI Engine Direct SDK, and includes components that are required for the stable Diffusion end-to-end pipeline. It demonstrates the use of Qualcomm AI Engine direct software and hardware to run the Stable Diffusion models on a Windows on Snapdragon device.
## User Inputs
In [11]:
import numpy as np
# Any user defined prompt
user_prompt = "decorated modern country house interior, 8 k, light reflections"
# User defined seed value
user_seed = np.int64(1.36477711e+14)
# User defined step value, any integer value in {20, 50}
user_step = 20
# User define text guidance, any float value in [5.0, 15.0]
user_text_guidance = 7.5
# Error checking for user_seed
assert isinstance(user_seed, np.int64) == True,"user_seed should be of type int64"
# Error checking for user_step
assert isinstance(user_step, int) == True,"user_step should be of type int"
assert user_step == 20 or user_step == 50,"user_step should be either 20 or 50"
# Error checking for user_text_guidance
assert isinstance(user_text_guidance, float) == True,"user_text_guidance should be of type float"
assert user_text_guidance >= 5.0 and user_text_guidance <= 15.0,"user_text_guidance should be a float from [5.0, 15.0]"
Copy to clipboard
## Embedding Functions
Expected execution time: 4mins
In [ ]:
import torch
from diffusers import UNet2DConditionModel
from diffusers.models.embeddings import get_timestep_embedding
# pre-load time embedding
time_embeddings = UNet2DConditionModel.from_pretrained('runwayml/stable-diffusion-v1-5',
subfolder='unet', cache_dir='./cache/diffusers').time_embedding
def get_time_embedding(timestep):
timestep = torch.tensor([timestep])
t_emb = get_timestep_embedding(timestep, 320, True, 0)
emb = time_embeddings(t_emb).detach().numpy()
return emb
Copy to clipboard
## Tokenizer
In [13]:
import numpy as np
from tokenizers import Tokenizer
# Define Tokenizer output max length (must be 77)
tokenizer_max_length = 77
# Initializing the Tokenizer
tokenizer = Tokenizer.from_pretrained("openai/clip-vit-base-patch32")
# Setting max length to tokenizer_max_length
tokenizer.enable_truncation(tokenizer_max_length)
tokenizer.enable_padding(pad_id=49407, length=tokenizer_max_length)
def run_tokenizer(prompt):
# Run Tokenizer encoding
token_ids = tokenizer.encode(prompt).ids
# Convert tokens list to np.array
token_ids = np.array(token_ids, dtype=np.float32)
return token_ids
Copy to clipboard
## Scheduler
In [14]:
import numpy as np
import torch
from diffusers import DPMSolverMultistepScheduler
# Initializing the Scheduler
scheduler = DPMSolverMultistepScheduler(num_train_timesteps=1000, beta_start=0.00085,
beta_end=0.012, beta_schedule="scaled_linear")
# Setting up user provided time steps for Scheduler
scheduler.set_timesteps(user_step)
def run_scheduler(noise_pred_uncond, noise_pred_text, latent_in, timestep):
# Convert all inputs from NHWC to NCHW
noise_pred_uncond = np.transpose(noise_pred_uncond, (0,3,1,2)).copy()
noise_pred_text = np.transpose(noise_pred_text, (0,3,1,2)).copy()
latent_in = np.transpose(latent_in, (0,3,1,2)).copy()
# Convert all inputs to torch tensors
noise_pred_uncond = torch.from_numpy(noise_pred_uncond)
noise_pred_text = torch.from_numpy(noise_pred_text)
latent_in = torch.from_numpy(latent_in)
# Merge noise_pred_uncond and noise_pred_text based on user_text_guidance
noise_pred = noise_pred_uncond + user_text_guidance * (noise_pred_text - noise_pred_uncond)
# Run Scheduler step
latent_out = scheduler.step(noise_pred, timestep, latent_in).prev_sample.numpy()
# Convert latent_out from NCHW to NHWC
latent_out = np.transpose(latent_out, (0,2,3,1)).copy()
return latent_out
# Function to get timesteps
def get_timestep(step):
return np.int32(scheduler.timesteps.numpy()[step])
Copy to clipboard
## Inference using Qualcomm AI Engine Direct
In [48]:
import numpy as np
import os
import shutil
# Define QNN binaries path
QNN_binaries_path = 'qnn_assets\\QNN_binaries'
# Define generic qnn-net-run block
def run_qnn_net_run(model_context, input_data_list):
# Define tmp directory path for intermediate artifacts
tmp_dirpath = os.path.abspath('tmp')
os.makedirs(tmp_dirpath, exist_ok=True)
# Dump each input data from input_data_list as raw file
# and prepare input_list_filepath for qnn-net-run
input_list_text = ''
for index, input_data in enumerate(input_data_list):
# Create and dump each input into raw file
raw_file_path = f'{tmp_dirpath}/input_{index}.raw'
input_data.tofile(raw_file_path)
# Keep appending raw_file_path into input_list_text for input_list_filepath file
input_list_text += raw_file_path + ' '
# Create input_list_filepath and add prepared input_list_text into this file
input_list_filepath = f'{tmp_dirpath}/input_list.txt'
with open(input_list_filepath, 'w') as f:
f.write(input_list_text)
# Execute qnn-net-run on shell
!{QNN_binaries_path}\qnn-net-run.exe --retrieve_context {model_context} --backend {QNN_binaries_path}/QnnHtp.dll \
--input_list {input_list_filepath} --output_dir {tmp_dirpath} > {tmp_dirpath}/log.txt
# Read the output data generated by qnn-net-run
output_data = np.fromfile(f'{tmp_dirpath}/Result_0/output_1.raw', dtype=np.float32)
# Delete all intermediate artifacts
shutil.rmtree(tmp_dirpath)
return output_data
# Define models context path
models_context_path = 'qnn_assets\\stable_diffusion_models'
# qnn-net-run for text encoder
def run_text_encoder(input_data):
output_data = run_qnn_net_run(f'{models_context_path}\\text_encoder\\text_encoder.serialized.bin', [input_data])
# Output of Text encoder should be of shape (1, 77, 768)
output_data = output_data.reshape((1, 77, 768))
return output_data
# qnn-net-run for U-Net
def run_unet(input_data_1, input_data_2, input_data_3):
output_data = run_qnn_net_run(f'{models_context_path}\\unet\\unet.serialized.bin', [input_data_1, input_data_2, input_data_3])
# Output of UNet should be of shape (1, 64, 64, 4)
output_data = output_data.reshape((1, 64, 64, 4))
return output_data
# qnn-net-run for VAE
def run_vae(input_data):
output_data = run_qnn_net_run(f'{models_context_path}\\vae_decoder\\vae_decoder.serialized.bin', [input_data])
# Convert floating point output into 8 bits RGB image
output_data = np.clip(output_data*255.0, 0.0, 255.0).astype(np.uint8)
# Output of VAE should be of shape (512, 512, 3)
output_data = output_data.reshape((512, 512, 3))
return output_data
Copy to clipboard
## Execute the Stable Diffusion pipeline
Expected execution time: 3mins
In [ ]:
# Run Tokenizer
uncond_tokens = run_tokenizer("")
cond_tokens = run_tokenizer(user_prompt)
# Run Text Encoder on Tokens
uncond_text_embedding = run_text_encoder(uncond_tokens)
user_text_embedding = run_text_encoder(cond_tokens)
# Initialize the latent input with random initial latent
random_init_latent = torch.randn((1, 4, 64, 64), generator=torch.manual_seed(user_seed)).numpy()
latent_in = random_init_latent.transpose((0, 2, 3, 1)).copy()
# Run the loop for user_step times
for step in range(user_step):
print(f'Step {step} Running...')
# Get timestep from step
timestep = get_timestep(step)
# Run U-net for const embeddings
unconditional_noise_pred = run_unet(latent_in, get_time_embedding(timestep), uncond_text_embedding)
# Run U-net for user text embeddings
conditional_noise_pred = run_unet(latent_in, get_time_embedding(timestep), user_text_embedding)
# Run Scheduler
latent_in = run_scheduler(unconditional_noise_pred, conditional_noise_pred, latent_in, timestep)
# Run VAE
output_image = run_vae(latent_in)
Copy to clipboard
In [ ]:
from PIL import Image
from IPython.display import display
# Display the generated output
display(Image.fromarray(output_image, mode="RGB"))
Copy to clipboard
**Example result**
# Final comments
Congratulations!
You have successfully worked through all the steps needed to execute the Stable Diffusion model using the Qualcomm AI Engine Direct SDK on a Windows on Snapdragon device.
Copyright (c) 2023 Qualcomm Technologies, Inc. and/or its subsidiaries.
**Parent Topic:** [Qualcomm AI Engine Direct Model Execution](https://docs.qualcomm.com/doc/80-64748-1/topic/model_execution.html)
Last Published: Apr 09, 2024
[Previous Topic
Qualcomm AI Engine Direct Model Execution](https://docs.qualcomm.com/bundle/publicresource/80-64748-1/topics/model_execution.md) [Next Topic
Qualcomm AI Engine Direct Model Execution on Android on Snapdragon](https://docs.qualcomm.com/bundle/publicresource/80-64748-1/topics/model_execution_android.md)