# Debug overview A subsystem processes independently within its own execution environment on the Qualcomm^®^ SoC. This section describes the tools, sample logs, and troubleshooting methods for impacted subsystems. Understanding the discrepancies in subsystems and methods for examining errors is useful for diagnosing and troubleshooting such anomalies. The following table lists the important subsystems on the SoC. Table: Subsystems on SoC | Subsystem | Description | | --- | --- | | Application processor subsystem (APSS) | This primary subsystem executes the Qualcomm^®^ Linux^®^ kernel as the high-level OS (HLOS). | | Application digital signal processor (aDSP) | This subsystem, also known as the low-power audio subsystem (LPASS), handles digital signal processing tasks, such as audio encoding/decoding, and voice recognition. | | Compute digital signal processor (cDSP) | This subsystem performs compute-intensive tasks, such as neural network-related calculations. | | Qualcomm^®^ Trusted Execution Environment (TEE)/TrustZone (TZ) | This subsystem performs secure operations and leverages the Arm^®^ TrustZone^®^ architecture. For more information, see [Qualcomm Linux Security Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-80022-11/security-home-page.html). | | Wireless local area network processor subsystem (WPSS) | This subsystem connects to a Wi-Fi network, transmits and receives data packets, handles security protocols, and ensures stable network performance. For more information, see [Qualcomm Linux Wi-Fi Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-80022-14/wifi_landing_page.html). | | Always-on Processor (AOP) | This subsystem regulates the power for the device. | Note The term non-HLOS refers to software for all subsystems other than the application processor. To understand the boot flow of the subsystems, see [Qualcomm Linux Boot Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-80022-4/landing-page.html). Note See [Hardware SoCs](https://docs.qualcomm.com/bundle/publicresource/topics/80-80022-115/soc.html) that are supported on Qualcomm^®^ Linux^®^. ## Debug workflow Identify the subsystem where the issue first occurs, as the Qualcomm SoC has many subsystems. This helps you debug the correct subsystem. The application processor, as the primary subsystem, detects when other subsystems crash. For example, if the aDSP processor subsystem crashes, the kernel log captures the subsystem restart (SSR) crash error log. Therefore, to identify the subsystem that needs debugging, first verify the kernel debug messages. The following figure shows the workflow to identify errors and the components you need to debug. **Figure: Workflow to identify the impacted subsystem** When the kernel dmesg logs show no panic signatures, verify the Qualcomm TEE diagnostic (diag) logs for errors such as nonsecure watchdog bite and network-on-chip (NoC) errors. The following sections describe example kernel messages that indicate kernel panic, bugs, and subsystem crash issues. ### Identify kernel panic and bugs You can determine from the kernel dmesg logs whether the reset was a kernel panic or a bug. The following example logs indicate a kernel panic or a bug: - Panic (Pattern: General error) 18.800936: <6> Kernel panic - not syncing: Fatal exception 18.800938: <6> SMP: stopping secondary CPUs 18.800947: <6> CPU0: stopping Kernel panic - not syncing: Apps watchdog Bark received! - Bug (Pattern 1: Forced crash caused by memory corruption) 12.899532: <6> BUG kmalloc-128 (Not tainted): Redzone overwritten 12.905418: <6> ----------------------------------------------------------------------------- - Bug (Pattern 2: Forced crash from driver) 320.510769: <6> ------------[ cut here ]------------ 320.510781: <6> kernel BUG at /local/mnt/workspace/lnxbuild/project/trees\_in\_use/free\_tree\_platform\_manifest\_refs\_tags/drivers/platform/msm/ipa/ipa\_v3/ipa\_qmi\_service.c:955! - Bug (Pattern 3: Forced crash due to lockup issue) [ 180.993861] BUG: spinlock lockup suspected on CPU#0, swapper/0/0 [ 180.993883] lock: stop\_lock+0x0/0x18, .magic: dead4ead, .owner: swapper/6/0, .owner\_cpu: 6 [ 181.015629] Causing a watchdog bite! Note Most bugs typically result in a kernel panic. For more information about debugging kernel issues, see [Qualcomm Linux Kernel Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-80022-3/kernel_landing_page.html). ### Identify subsystem crash issues You can verify the kernel dmesg logs to determine whether the restart was due to a subsystem crash. For example, the following log indicates that the aDSP subsystem crashed, according to the `sc7280.dtsi` file (node: `remoteproc_adsp: remoteproc@3700000`): 0x000000000A27652C | 5198.790423: qcom\_q6v5\_pas 3700000.remoteproc: fatal error received: err\_inject\_crash.c:413:Crash injected via Diag 0x000000000A276689 | 5198.801061: remoteproc remoteproc2: crash detected in 3000000.remoteproc: type fatal error 0x000000000A2767A1 | 5198.809602: remoteproc remoteproc2: handling crash #1 in 3000000.remoteproc 0x000000000A27688E | 5198.816837: remoteproc remoteproc2: recovering 3000000.remoteproc 0x000000000A276971 | 5198.823784: qcom\_q6v5\_pas 8a00000.remoteproc: subsystem event rejected ### Identify system issues from Qualcomm TEE logs This feature is available to licensed developers with authorized access. For sample logs that indicate errors in the Qualcomm TEE, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-80022-12A/identify_subsystem_to_be_debugged.html#identify_and_debug_errors_in_tz). If the Qualcomm TEE diag logs show no error, the issue may be due to a secure watchdog bite. For more information about watchdog issues, see [Hardware reset](https://docs.qualcomm.com/doc/80-80022-12/topic/reset_hardware.html#reset-hardware). For more information about the debugging issues in Qualcomm TEE, see [Qualcomm Linux Security Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-80022-11/debug.html). ## Debug methods The following are the two approaches for debugging: - **On-target debugging** On-target debugging resolves software issues. This approach allows access to most information directly from the live device. Connect the Linux host computer to the live device using SSH. To set up SSH, see [Sign in using SSH](https://docs.qualcomm.com/bundle/publicresource/topics/80-80022-254/how_to.html#use-ssh). For more information about various on-target debug features, see [On-target debug features](https://docs.qualcomm.com/doc/80-80022-12/topic/on_target_debug_features.html#on-target-debug-features). - **Off-target debugging** This approach uses logs instead of an actual device, making it an efficient method of debugging. You can debug using memory dumps or logging tools and various types of logs for offline debugging. Although the RAM dump captures most of the memory regions, it includes only limited hardware register information, so debugging a hardware-related issue can be a challenge. For more information about various off-target debug features, see [Off-target debug features](https://docs.qualcomm.com/doc/80-80022-12/topic/off_target_debug_features.html#off-target-debug-features). ### On-target debug features You can use the following debug features on the device at runtime: - [Perf utility](https://docs.qualcomm.com/doc/80-80022-12/topic/on_target_debug_features.html#section-tmb-3v5-41c) - [Subsystem restart](https://docs.qualcomm.com/doc/80-80022-12/topic/on_target_debug_features.html#section-o5w-vfb-pzb) - [Force subsystem reset](https://docs.qualcomm.com/doc/80-80022-12/topic/on_target_debug_features.html#section-gdz-gjb-pzb) #### Perf utility The Perf utility in Linux facilitates performance analysis and profiling. The Linux kernel includes Perf utility in the `tools/perf` directory. You can use the Perf utility to debug various aspects of system behavior, including CPU performance counters, tracepoints, kprobes, and uprobes (for dynamic tracing). The following table lists the key features of the Perf utility. Table: Key features of Perf utility | Feature | Description | | --- | --- | | CPU performance counters | These CPU hardware registers track events, such as executed instructions, cache misses, and branch mispredictions. These events form the basis for profiling applications and identifying performance bottlenecks. | | Tracepoints | You can place these tracepoints at logical locations in the code, such as system calls, network events, and file system operations. These tracepoints provide information,
such as timestamps and stack traces, with minimal overhead. | | Dynamic tracing | This feature dynamically creates tracepoints using the kprobes and uprobes frameworks, allowing tracing in both kernel space and user space. | For more information about the Perf utility, see the following resources: - [Perf main page](https://perf.wiki.kernel.org/index.php/Main_Page) - [HOWTO](https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md) - [CoreSight](https://docs.kernel.org/trace/coresight/index.html) - [CoreSight - Perf](https://docs.kernel.org/trace/coresight/coresight-perf.html) #### Subsystem restart Subsystems can restart independently when they crash without requiring a device restart called subsystem restart (SSR). The SSR feature uses the remoteproc framework available in the Linux kernel. To enable a full memory dump for debugging purposes, disable the SSR feature by default. Note It's recommended to use the SSR feature on commercial devices and not during the development phase. For more information about the SSR feature, see [Configure the remoteprocessor subsystems](https://docs.qualcomm.com/bundle/publicresource/topics/80-80022-3/remoteproc-overview.html). Enable the SSR feature to generate the SSR dump on a subsystem restart. Consider using the SSR dump for debugging, as its size is smaller than the full memory dump. However, you need the entire RAM dump to debug some subsystem crashes. For more information about how to enable and capture subsystem dumps, see [Capture the subsystem dumps](https://docs.qualcomm.com/doc/80-80022-12/topic/subsystem_ram_dumps.html#subsystem-ram-dumps). #### Force subsystem reset Use this feature to debug use cases that require restarting a subsystem. Note This feature is available to licensed developers with authorized access. To force a subsystem reset, you can run the `diag` command using the QXDM Professional^™^ tool. For more information about QXDM Professional commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-80022-12A/debug-non-hlos-subsystems.html#qxdm_professional). ### Off-target debug features You can use RAM dump or QXDM Professional for offline debugging. #### RAM dump The RAM dump is a snapshot of the entire memory at the time of failure. You can analyze the RAM dump using various tools, including Qualcomm^®^ Crash Analysis Portal (QCAP), RAMParser, Lauterbach TRACE32 simulator, and crash utility. For more information about how to collect and parse RAM dumps, see [Collect and parse RAM dump](https://docs.qualcomm.com/doc/80-80022-12/topic/collect_and_parse_ram_dumps.html#collect-and-parse-ram-dumps). #### QXDM Professional Use QXDM Professional to debug various subsystems. This tool is available to licensed developers with authorized access. For more information about QXDM Professional commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-80022-12A/debug-non-hlos-subsystems.html#qxdm_professional). ## Common system issues You can categorize system issues into one of the following types: > > > - [Linux kernel-space issues](https://docs.qualcomm.com/doc/80-80022-12/topic/debugging_linux_kernel.html#debugging-linux-kernel) > - [System issues](https://docs.qualcomm.com/doc/80-80022-12/topic/general_system_debugging.html#general-system-debugging) > > > > > > > > > - [Watchdog issues](https://docs.qualcomm.com/doc/80-80022-12/topic/watchdog_timeout.html#watchdog-timeout) > > - [Bus hang and timeout issues](https://docs.qualcomm.com/doc/80-80022-12/topic/general_system_debugging.html#erroneous-transaction-on-bus-error-and-timeout) > > - [Hardware reset issues](https://docs.qualcomm.com/doc/80-80022-12/topic/reset_hardware.html#reset-hardware) ## Miscellaneous issues You may run into the following miscellaneous issues: - Device-freeze issues due to software bugs - Linux application-related issues - Random resets due to hardware issues related to the PCB ## Next steps > > > - To debug Linux application-related issues, see [Debug Linux user-space issues](https://docs.qualcomm.com/doc/80-80022-12/topic/using_open_source_debug_tools.html#using-open-source-debug-tools). > - To debug Linux kernel-related issues, see [Debug Linux kernel-space issues](https://docs.qualcomm.com/doc/80-80022-12/topic/debugging_linux_kernel.html#debugging-linux-kernel). Last Published: May 14, 2026 [Previous Topic Debug documentation](https://docs.qualcomm.com/bundle/publicresource/80-80022-12/topics/landing_page.md) [Next Topic Debug Linux user-space issues](https://docs.qualcomm.com/bundle/publicresource/80-80022-12/topics/using_open_source_debug_tools.md)