# Linux ARM64 Genie Service Setup

## Purpose

This document defines a process to package, stage, start, validate, and troubleshoot Linux ARM64 Genie Service deployments across supported devices.

## Required Inputs

- <DEVICE\_USER>, <DEVICE\_HOST>: target SSH credentials

## Supported Devices

| Platform | SoC | SoC ID | Architecture | DSP Version |
| --- | --- | --- | --- | --- |
| LeMans IoT | QCS9075 | 676 | aarch64-oe-linux-gcc11.2 | V73 |
| Monza IoT | QCS8275 | 675 | aarch64-oe-linux-gcc11.2 | V75 |

## Preparing Genie Service Test Package

### Step 1: Build Genie Service Test Package

1. Create test package directory

> 
> 
> mkdir -p genie_service/lib genie_service/config genie_service/dsp
>         Copy to clipboard
2. Copy binary from Genie release:

> 
> 
> cp <release>/bin/aarch64-oe-linux-gcc11.2/genie-service genie_service/genie-service
>         Copy to clipboard
3. Copy Genie library dependencies from Genie release:

> 
> 
> cp <release>/lib/aarch64-oe-linux-gcc11.2/libGenie.so <release>/lib/aarch64-oe-linux-gcc11.2/libGenieService.so <release>/lib/aarch64-oe-linux-gcc11.2/libQnnHtp.so <release>/lib/aarch64-oe-linux-gcc11.2/libQnnHtpNetRunExtensions.so <release>/lib/aarch64-oe-linux-gcc11.2/libQnnHtp${DSP_VER}Stub.so <release>/lib/aarch64-oe-linux-gcc11.2/libQnnSystem.so genie_service/lib/
>         Copy to clipboard
4. Copy DSP library dependencies from Genie release:

> 
> 
> cp <release>/lib/hexagon-${DSP_VER}/lib/libQnnHtp${DSP_VER}Skel.so genie_service/dsp
>         Copy to clipboard
5. Copy Service Config:

> 
> 
> cp <release>/examples/Genie/OpenAI/config/genie_openai_service_config.yaml genie_service/config/
>         cp <release>/examples/Genie/OpenAI/config/genie_service.service genie_service/config/
>         Copy to clipboard
> 
>     - Note: Service config can be modified to use preferred models
6. Finalize:

> 
> 
> chmod +x genie_service/genie-service
>         chmod 755 genie_service/lib/*.so genie_service/dsp/*.so
>         tar -czf genie_service.tgz -C genie_service .
>         Copy to clipboard

### Step 2: Stage Package and Models on Device

1. Copy tarball to device

> 
> 
> scp genie_service.tgz <DEVICE_USER>@<DEVICE_HOST>:/tmp/genie_service.tgz
>         Copy to clipboard
2. Unzip tarball on device

> 
> 
> ssh <DEVICE_USER>@<DEVICE_HOST> 'rm -rf /tmp/genie_service && mkdir -p /tmp/genie_service && tar -xzf /tmp/genie_service.tgz -C /tmp/genie_service'
>         Copy to clipboard
3. Create model folder

> 
> 
> ssh <DEVICE_USER>@<DEVICE_HOST> 'mkdir -p ${TARGET_INSTALL_PATH}/models'
>         Copy to clipboard
4. Copy model files to device

> 
> 
> rsync -av <model_dir>/ <DEVICE_USER>@<DEVICE_HOST>:${TARGET_INSTALL_PATH}/models/<model_dir>/
>         Copy to clipboard
> 
>     - Note: Must copy Genie compatible models, including JSON configs and tokenizer JSON.
>     - Note: Ensure that the paths in the model config files are valid paths on device, ex. /tmp instead of /data/local/tmp.

### Step 3: Clear Stale Services

1. Before every run, take care to stop old genie-service listeners on chosen HTTP port.

> 
> 
> ssh <DEVICE_USER>@<DEVICE_HOST> 'pkill -x genie-service || true; for pid in $(pgrep -f "genie-service --config" || true); do kill "$pid" || true; done; sleep 2; pgrep -af "genie-service " || true; ss -ltnp 2>/dev/null | grep 18080 || true'
>         Copy to clipboard
> 
>     - Note: Multiple listeners listening on the same port can route requests to the wrong runtime.

### Step 4: Start New Service

1. On device:

> 
> 
> cd /tmp/genie_service
>         LD_LIBRARY_PATH=/tmp/genie_service/lib ADSP_LIBRARY_PATH=/tmp/genie_service/dsp /tmp/genie_service/genie-service --config /tmp/genie_service/config/genie_openai_service_config.yaml > /tmp/genie_service/service.log 2>&1 < /dev/null &
>         Copy to clipboard
> 
>     - Verify process, listening port, and recent logs.

### Step 5: Execute Python Test App

1. Update model IDs per test case in `OpenAIClientApp_test_suite.json` to match those configured on the device.

> 
> 
> - Note: Do not edit `OpenAIClientApp.py`.
2. From `<release>` directory, run:

> 
> 
> env NO_PROXY=<DEVICE_HOST>,127.0.0.1,localhost OCA_BASE_URL=http://<DEVICE_HOST>:18080 OCA_SKIP_MULTIMODAL=true OCA_SKIP_ADMIN=true python3 <release>/examples/Genie/OpenAI/OpenAIClientApp/OpenAIClientApp.py 2>&1 | tee <RESULTS_DIR>/<RUN_NAME>_python_test.log
>         Copy to clipboard
> 
>     - Exit code `0` means all tests passed or were skipped; exit code `1` means one or more tests failed or the service was unreachable.

#### Example Test Cases

In the current version of the Genie Service and Test App, we support the following API endpoints:

- POST /v1/chat/completions
- POST /v1/embeddings
- GET /v1/models

The following examples illustrate test cases for available APIs. Examples for all test cases are defined in `OpenAIClientApp_test_suite.json`; update the `model` / `model_id` fields to match the model IDs configured in the service.

**models\_list** — /v1/models — List all loaded models

> 
> 
> {
>       "id": "<test case id, e.g. list_models>",
>       "name": "<test case name, e.g. GET /v1/models returns at least one model>",
>       "type": "models_list",
>       "request": {},
>       "expected": {
>         "min_model_count": 1
>       }
>     }
>     Copy to clipboard

**models\_retrieve** — /v1/models/{model} — Retrieve a specific model by ID

> 
> 
> {
>       "id": "<test case id, e.g. retrieve_my_model>",
>       "name": "<test case name, e.g. GET /v1/models/meta-llama/llama3.2-3b>",
>       "type": "models_retrieve",
>       "request": {
>         "model": "<model ID to retrieve, e.g. meta-llama/llama3.2-3b>"
>       },
>       "expected": {
>         "object": "model",
>         "id_matches_request": true
>       }
>     }
>     Copy to clipboard

**chat\_completion** — /v1/chat/completions — Non-streaming chat completion

> 
> 
> {
>       "id": "<test case id, e.g. basic_chat>",
>       "name": "<test case name, e.g. Basic chat completion>",
>       "type": "chat_completion",
>       "request": {
>         "model": "<model ID to use, e.g. meta-llama/llama3.2-3b>",
>         "messages": [
>           {
>             "role": "<role to use, e.g. user>",
>             "content": "<message content, e.g. Say hello.>"
>           }
>         ],
>         "max_tokens": 20,
>         "temperature": 0.7
>       },
>       "expected": {
>         "object": "chat.completion",
>         "choices_count": 1,
>         "choice_role": "<expected role, e.g. assistant>",
>         "has_content": true,
>         "has_usage": true
>       }
>     }
>     Copy to clipboard

**chat\_completion\_stream** — /v1/chat/completions — Streaming chat completion

> 
> 
> {
>       "id": "<test case id, e.g. streaming_chat>",
>       "name": "<test case name, e.g. Streaming chat completion>",
>       "type": "chat_completion_stream",
>       "request": {
>         "model": "<model ID to use, e.g. meta-llama/llama3.2-3b>",
>         "messages": [
>           {
>             "role": "<role to use, e.g. user>",
>             "content": "<message content, e.g. Count to 5.>"
>           }
>         ],
>         "max_tokens": 30
>       },
>       "expected": {
>         "chunks_min": 1,
>         "full_text_not_empty": true
>       }
>     }
>     Copy to clipboard

**embedding** — /v1/embeddings — Generate text embeddings

> 
> 
> {
>       "id": "<test case id, e.g. single_embedding>",
>       "name": "<test case name, e.g. Single string embedding>",
>       "type": "embedding",
>       "request": {
>         "model": "<model ID to use, e.g. BAAI/bge-large-en>",
>         "input": "<input text to embed, e.g. Hello world>",
>         "encoding_format": "<encoding format to output, e.g. float>"
>       },
>       "expected": {
>         "embedding_count": 1,
>         "embedding_dim_min": 128,
>         "has_usage": true
>       }
>     }
>     Copy to clipboard

**parallel** — Concurrent sub-requests

> 
> 
> {
>       "id": "<test case id, e.g. parallel_chat_completions>",
>       "name": "<test case name, e.g. 4 concurrent chat completions>",
>       "type": "parallel",
>       "request": {
>         "sub_tests": [
>           {
>             "id": "<test case id, e.g. req_1>",
>             "type": "chat_completion",
>             "request": {
>               "model": "<model ID to use, e.g. meta-llama/llama3.2-3b>",
>               "messages": [{ "role": "user", "content": "Say: alpha" }],
>               "max_tokens": 10
>             },
>             "expected": { "has_content": true }
>           },
>           {
>             "id": "<test case id, e.g. req_2>",
>             "type": "chat_completion",
>             "request": {
>               "model": "<model ID to use, e.g. meta-llama/llama3.2-3b>",
>               "messages": [{ "role": "user", "content": "Say: beta" }],
>               "max_tokens": 10
>             },
>             "expected": { "has_content": true }
>           },
>           {
>             "id": "<test case id, e.g. req_3>",
>             "type": "chat_completion",
>             "request": {
>               "model": "<model ID to use, e.g. meta-llama/llama3.2-3b>",
>               "messages": [{ "role": "user", "content": "Say: gamma" }],
>               "max_tokens": 10
>             },
>             "expected": { "has_content": true }
>           },
>           {
>             "id": "<test case id, e.g. req_4>",
>             "type": "chat_completion",
>             "request": {
>               "model": "<model ID to use, e.g. meta-llama/llama3.2-3b>",
>               "messages": [{ "role": "user", "content": "Say: delta" }],
>               "max_tokens": 10
>             },
>             "expected": { "has_content": true }
>           }
>         ]
>       },
>       "expected": {
>         "all_pass": true,
>         "max_workers": 4
>       }
>     }
>     Copy to clipboard

## Genie Service Configuration

The Genie Service can be customized by modifying the service configuration file at /tmp/genie\_service/config/genie\_openai\_service\_config.yaml. The following is a breakdown of the sections in the config, the default configuration, and what modifications are possible:

### Version

> 
> 
> # Genie OpenAI Service Configuration
>     version: "1.0"
>     Copy to clipboard
> 
> - Version should not be modified manually besides during major Genie version updates

### HTTP Server

> 
> 
> # ──────────────────────────────────────────────────────────────────────────────
>     # HTTP Server
>     # ──────────────────────────────────────────────────────────────────────────────
>     server:
>       host: "0.0.0.0"              # Bind address
>       port: 8080                   # Listen port
>       thread_pool_size: 16         # HTTP threads (one per active request)
>       request_timeout_seconds: 120 # Max wait time for a single inference
>     Copy to clipboard
> 
> - HTTP Server parameters can be configured based on need, ensure that only one service is running on a given port to avoid conflicts and requests timing out.

### Resource Limits

> 
> 
> # ──────────────────────────────────────────────────────────────────────────────
>     # Resource Limits
>     # ──────────────────────────────────────────────────────────────────────────────
>     # memory_budget_mb: Maximum memory allowed for the service process.
>     #   - Checked before each model load
>     #   - If current usage exceeds threshold, evict LRU or reject load
>     #   - Allows flexible model combinations (many small or few large)
>     #   - Set to 0 to disable (no memory guard, load until NPU OOM)
>     limits:
>       memory_budget_mb: 12000 # Total memory budget in MB (0 = unlimited)
>       eviction_policy: lru    # lru | none
>     Copy to clipboard
> 
> - Resource limits can be configured for this service to allow for flexible combinations of models loaded. Possible configuration options are listed above.

### Watchdog

> 
> 
> # ──────────────────────────────────────────────────────────────────────────────
>     # Watchdog
>     # ──────────────────────────────────────────────────────────────────────────────
>     # Sends sd_notify(WATCHDOG=1) to systemd periodically.
>     # Also monitors worker threads for stuck inference (e.g., SDK hang).
>     # If a worker exceeds stuck_threshold_seconds with no progress,
>     # a warning is logged and the current request can be force-cancelled.
>     watchdog:
>       enabled: true
>       interval_seconds: 10         # sd_notify frequency
>       stuck_threshold_seconds: 300 # 5 min = likely stuck
>     Copy to clipboard
> 
> - Watchdog can be configured to monitor for and cancel stuck Genie SDK calls.

### Models

> 
> 
> models:
>     Copy to clipboard
> 
> - The models section contains the configuration for the individual models supported in this instance of the Genie service. Currently the service supports “llm”, “embedding”, and “lmm” model types:
> 
> 
>     LLM
> 
> 
> 
> > 
> > 
> > # ════════════════════════════════════════════════════════════════════════════
> >         # LLM (Text Generation)
> >         # ════════════════════════════════════════════════════════════════════════════
> >         - model_id: "meta-llama/llama3.2-3b"
> >           model_type: llm
> >           provider: genie
> >           auto_load: true
> >           memory_estimate_mb: 4096
> >           # Worker queue: requests waiting for this model's NPU turn.
> >           # When full, server returns HTTP error
> >           queue:
> >             max_depth: 16
> >           # Genie SDK model configuration.
> >           # config_file points to the Genie JSON that defines model architecture,
> >           # weights path, tokenizer, context length, etc.
> >           # genie_log_level controls Genie SDK internal logging (separate from service log).
> >           provider_config:
> >             config_file: /data/local/tmp/qaior-runtime/models/llm-llama3.2/genie_config.json
> >             genie_log_level: verbose
> >           # Default inference parameters.
> >           # Used when the client doesn't specify them in the request.
> >           # Client-provided values always override these.
> >           defaults:
> >             temperature: 1.0
> >             top_p: 1.0
> >             max_tokens: 2048
> >             stop_sequences:
> >               - "<|eot_id|>"
> >               - "<|eom_id|>"
> >           # LoRA adapter configuration (optional).
> >           # Applied at model load time. Currently not switchable per-request.
> >           # name: must match lora::adapters::name in the Genie JSON config_file.
> >           # alpha: controls adapter strength. 0.0 = disabled, 1.0 = full strength.
> >           lora:
> >             name: default_adapter
> >             alpha:
> >               - name: lora_alpha
> >                 value: 0.0
> >         
> >           # NPU performance profile.
> >           # Controls power/performance tradeoff on Hexagon DSP.
> >           # Values: burst | sustained_high_performance | high_performance |
> >           #         balanced | low_balanced | high_power_saver |
> >           #         low_power_saver | power_saver | extreme_power_saver
> >           performance:
> >             htp_perf_profile: balanced
> >         
> >           # Prompt formatting
> >           prompting:
> >             default_system_prompt: |
> >               You are a helpful, respectful, and honest assistant.
> >               Always answer as helpfully as possible.
> >             system_header: "<|start_header_id|>system<|end_header_id|>\n\n"
> >             system_footer: "<|eot_id|>"
> >             user_header: "<|start_header_id|>user<|end_header_id|>\n\n"
> >             user_footer: "<|eot_id|>"
> >             assistant_header: "<|start_header_id|>assistant<|end_header_id|>\n\n"
> >             assistant_footer: "<|eot_id|>"
> >         
> >           # ──────────────────────────────────────────────────────────────────────────────
> >           # Session
> >           # ──────────────────────────────────────────────────────────────────────────────
> >           # Session management strategy for managing conversation context
> >           # Values: engine_sharing, save_restore, single
> >           session_strategy: single
> >         Copy to clipboard
> 
> 
>     - model\_id can be chosen by the user but must be used consistently on both service and client side.
>     - model\_type must be llm and provider must be genie
>     - auto\_load defines whether the model is loaded during service initialization or during the first client invocation. true means load during service init, false means load during first client invocation
>     - queue-&gt;max\_depth defines how many requests for this model, from all clients, can be queued by the service. Any additional requests would be rejected by the service.
>     - provider\_config-&gt;config\_file must be a json file containing the Genie configuration for this model.
>     - provider\_config-&gt;genie\_log\_level is the model-specific logging level to be used when this model is invoked.
>     - defaults contains default inference parameters for cases where the client does not provide preferred parameters. temperature must be non-zero to avoid inference calls hanging.
>     - lora is optional, but if used lora-&gt;name must match the lora name defined in the Genie config file for this model. If a lora is defined it will be used for all inference calls, and cannot be turned on/off between calls.
>     - performance-&gt;htp\_perf\_profile allows the user to optionally configure the performance mode for inference calls. The performance mode chosen here will apply to all inference calls and cannot be customized between calls. Accepted values are burst, sustained\_high\_performance, high\_performance, balanced, low\_balanced, high\_power\_saver, low\_power\_saver, power\_saver, extreme\_power\_saver.
>     - prompting allows for configuring the default system prompt, system headers/footers, user headers/footers, and assistant headers/footers. These can be modified based on preferences and the model itself.
> 
> 
> 
>     Embedding
> 
> 
> 
> > 
> > 
> > # ════════════════════════════════════════════════════════════════════════════
> >         # Embedding
> >         # ════════════════════════════════════════════════════════════════════════════
> >         # Fully stateless. No session, no prompting, no streaming.
> >         # Each input string → one embedding vector.
> >         # Used by RAG pipelines, semantic search, etc.
> >         - model_id: "bge-large-zh"
> >           model_type: embedding
> >           provider: genie
> >           auto_load: true
> >           memory_estimate_mb: 4096
> >           queue:
> >             max_depth: 32
> >           provider_config:
> >             config_file: /data/local/tmp/qaior-runtime/models/embedding_api/bge-large-htp.json
> >             genie_log_level: verbose
> >           performance:
> >             htp_perf_profile: balanced
> >         Copy to clipboard
> > 
> >     - model\_id can be chosen by the user but must be used consistently on both service and client side.
> >     - model\_type must be embedding and provider must be genie
> >     - auto\_load defines whether the model is loaded during service initialization or during the first client invocation. true means load during service init, false means load during first client invocation
> >     - queue-&gt;max\_depth defines how many requests for this model, from all clients, can be queued by the service. Any additional requests would be rejected by the service.
> >     - provider\_config-&gt;config\_file must be a json file containing the Genie configuration for this model.
> >     - provider\_config-&gt;genie\_log\_level is the model-specific logging level to be used when this model is invoked.
> >     - performance-&gt;htp\_perf\_profile allows the user to optionally configure the performance mode for inference calls. The performance mode chosen here will apply to all inference calls and cannot be customized between calls. Accepted values are burst, sustained\_high\_performance, high\_performance, balanced, low\_balanced, high\_power\_saver, low\_power\_saver, power\_saver, extreme\_power\_saver.
> 
> 
> 
>     LMM
> 
> 
> 
> > 
> > 
> > # ════════════════════════════════════════════════════════════════════════════
> >         # LMM (Vision Language Model)
> >         # ════════════════════════════════════════════════════════════════════════════
> >         # Multimodal: text + image input, text output. config_file is the combined
> >         # Genie pipeline config that wires LMM nodes together.
> >         #
> >         # image_preprocessing_config is required when the model's image encoder
> >         # expects an already-decoded, model-specific tensor input Omit it when
> >         # the encoder accepts raw encoded image bytes (JPEG/PNG) directly and decodes
> >         # them internally.
> >         - model_id: "internvl3.5-1b"
> >           model_type: lmm
> >           provider: genie
> >           auto_load: false
> >           session_strategy: single
> >           queue:
> >             max_depth: 4
> >           provider_config:
> >             config_file: /data/local/tmp/qaior-runtime/models/lmm-internvl3_5-1b/pipeline_config.json
> >             genie_log_level: info
> >           performance:
> >             htp_perf_profile: balanced
> >           # Optional: external image preprocessing config for LMM Model.
> >           image_preprocessing_config: /data/local/tmp/qaior-runtime/models/lmm-internvl3_5-1b/image_preprocessing_config.yaml
> >         
> >           # Prompt formatting template.
> >           # system_header/eot wrap the system turn; user_header/eot wrap the user turn;
> >           # assistant_header is appended as the generation primer after the last user turn.
> >           prompting:
> >             default_system_prompt: "You are a helpful visual assistant."
> >             system_header: "<|im_start|>system\n"
> >             system_footer: "<|im_end|>\n"
> >             user_header: "<|im_start|>user\n"
> >             user_footer: "<|im_end|>\n"
> >             assistant_header: "<|im_start|>assistant\n"
> >             assistant_footer: "<|im_end|>"
> >             # Empty → content_grouped placement (all images collected before user text).
> >             # Non-empty → content_inline placement with wrapper tokens around each image.
> >             image_header: ""
> >             image_footer: ""
> >         Copy to clipboard
> > 
> >     - model\_id can be chosen by the user but must be used consistently on both service and client side.
> >     - model\_type must be lmm and provider must be genie
> >     - auto\_load defines whether the model is loaded during service initialization or during the first client invocation. true means load during service init, false means load during first client invocation
> >     - queue-&gt;max\_depth defines how many requests for this model, from all clients, can be queued by the service. Any additional requests would be rejected by the service.
> >     - provider\_config-&gt;config\_file must be a json file containing the Genie configuration for this model.
> >     - provider\_config-&gt;genie\_log\_level is the model-specific logging level to be used when this model is invoked.
> >     - performance-&gt;htp\_perf\_profile allows the user to optionally configure the performance mode for inference calls. The performance mode chosen here will apply to all inference calls and cannot be customized between calls. Accepted values are burst, sustained\_high\_performance, high\_performance, balanced, low\_balanced, high\_power\_saver, low\_power\_saver, power\_saver, extreme\_power\_saver.
> >     - image\_preprocessing\_config is an optional configuration file to configure image pre-processing for this LMM model.
> >     - prompting allows for configuring the default system prompt, system headers/footers, user headers/footers, assistant headers/footers, and image headers/footers. These can be modified based on preferences and the model itself.

## Genie Model Configuration

Any models used by the Genie Service must include a Genie model configuration file alongside the model binaries. Here is an example Genie model configuration for reference, ensure that the tokenizer path, extensions path, and ctx-bins point to the correct model files:

Example Genie model configs can be found here: ${QAIRT\_SDK\_ROOT}/examples/Genie/configs

Last Published: Aug 06, 2026

[Previous Topic
Genie DLC](https://docs.qualcomm.com/bundle/publicresource/80-63442-10/topics/dlc.md) [Next Topic
Library](https://docs.qualcomm.com/bundle/publicresource/80-63442-10/topics/library.md)