# Debug overview
A subsystem processes independently within its own execution environment on the Qualcomm^®^ SoC. This section describes the tools, sample logs, and troubleshooting methods for impacted subsystems. Understanding the discrepancies in subsystems and methods for examining errors is useful for diagnosing and troubleshooting such anomalies.
The following table lists the important subsystems on the SoC.
Table: Subsystems on SoC
| Subsystem | Description |
| --- | --- |
| Application processor subsystem (APSS) | This primary subsystem executes the Qualcomm Linux kernel as the high-level OS (HLOS). |
| Application digital signal processor (aDSP) | This subsystem, known as the low-power audio subsystem (LPASS), handles digital signal processing tasks, such as audio encoding/decoding, and voice recognition. |
| Compute digital signal processor (cDSP) | This subsystem, known as the compute DSP, performs compute-intensive tasks, such as neural network-related calculations. |
| Qualcomm^®^ Trusted Execution Environment (TEE)/TrustZone (TZ) | This subsystem performs secure operations and leverages the Arm® TrustZone® architecture. For more information, see [Qualcomm Linux Security Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-11/security-home-page.html). |
| Wireless local area network processor subsystem (WPSS) | This subsystem connects to a Wi-Fi network, transmits and receives data packets, handles security protocols, and ensures stable network performance. For more information, see [Qualcomm Linux Wi-Fi Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-14/wifi_landing_page.html). |
| Always On Processor (AOP) | This subsystem regulates the power on the device. |
Software for all subsystems other than the application processor is referred to as non-HLOS.
To understand the boot flow of the subsystems, see [Qualcomm Linux Boot Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-4/landing-page.html).
Note
See [Hardware SoCs](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-115/soc.html) that Qualcomm^®^ Linux^®^ supports.
## Next steps
>
>
> - To identify errors and the components you need to debug, see [Debug workflow](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#identify-subsystem-to-be-debugged).
> - To know about different debugging techniques, see [Debug methods](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#debugging-approaches).
> - To access the information directly from the live device, see [On-target debug features](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#on-target-debug-features).
> - To learn how to use RAM dumps or QXDM Professional for offline debugging, see [Off-target debug features](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#off-target-debug-features).
> - To debug common system issues, see [Common system issues](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#common-errors).
> - To debug miscellaneous issues, see [Miscellaneous issues](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#miscellaneous-errors).
## Debug workflow
Identify the subsystem where the issue occurred first, as many subsystems are present on the Qualcomm SoC. This helps you debug the relevant subsystem.
The application processor, as the primary subsystem detects when other subsystems crash. For example, if the aDSP processor subsystem crashes, the kernel log captures the subsystem restart (SSR) crash error log. Therefore, to identify the subsystem that need debugging, first verify the kernel debug messages.
The following figure shows the workflow to identify errors and the components you need to debug.
**Figure : Workflow to identify the impacted subsystem**
When the kernel dmesg logs show no panic signatures, verify the Qualcomm TEE diagnostic (diag) logs for errors such as nonsecure watchdog bite and network-on-chip (NoC) errors. The following sections describe example kernel messages that indicate kernel panic, bugs, and subsystem crash issues.
### Identify kernel panic and bugs
You can determine from the kernel dmesg logs whether the reset was a kernel panic or a bug. The following are some example logs that indicate a kernel panic or a bug:
- Panic (Pattern: General error)
18.800936: <6> Kernel panic - not syncing: Fatal exception
18.800938: <6> SMP: stopping secondary CPUs
18.800947: <6> CPU0: stopping
Kernel panic - not syncing: Apps watchdog Bark received!
- Bug (Pattern 1: Forced crash due to memory corruption)
12.899532: <6> BUG kmalloc-128 (Not tainted): Redzone overwritten
12.905418: <6> -----------------------------------------------------------------------------
- Bug (Pattern 2: Forced crash from driver)
320.510769: <6> ------------[ cut here ]------------
320.510781: <6> kernel BUG at /local/mnt/workspace/lnxbuild/project/trees\_in\_use/free\_tree\_platform\_manifest\_refs\_tags/drivers/platform/msm/ipa/ipa\_v3/ipa\_qmi\_service.c:955!
- Bug (Pattern 3: Forced crash due to lockup issue)
[ 180.993861] BUG: spinlock lockup suspected on CPU#0, swapper/0/0
[ 180.993883] lock: stop\_lock+0x0/0x18, .magic: dead4ead, .owner: swapper/6/0, .owner\_cpu: 6
[ 181.015629] Causing a watchdog bite!
Note
Most bugs typically result in a kernel panic. For more information about debugging kernel issues, see [Qualcomm Linux Kernel Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-3/kernel_landing_page.html).
### Identify subsystem crash issues
You can verify the kernel dmesg logs to determine whether the restart was due to a subsystem crash. For example, the following log indicates that the aDSP subsystem crashed, according to the `qcm6490.dtsi` (node: remoteproc\_adsp: remoteproc@3000000).
0x000000000A27652C | 5198.790423: qcom\_q6v5\_pas 3000000.remoteproc: fatal error received: err\_inject\_crash.c:413:Crash injected via Diag
0x000000000A276689 | 5198.801061: remoteproc remoteproc2: crash detected in 3000000.remoteproc: type fatal error
0x000000000A2767A1 | 5198.809602: remoteproc remoteproc2: handling crash #1 in 3000000.remoteproc
0x000000000A27688E | 5198.816837: remoteproc remoteproc2: recovering 3000000.remoteproc
0x000000000A276971 | 5198.823784: qcom\_q6v5\_pas 8a00000.remoteproc: subsystem event rejected
### Identify system issues from Qualcomm TEE logs
This feature is available to licensed developers with authorized access. For sample logs that indicate errors in the Qualcomm TEE, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70022-12A/identify_subsystem_to_be_debugged.html#identify_and_debug_errors_in_tz).
If the Qualcomm TEE diag logs shows no error, the issue may be due to a secure watchdog bite issue. For more information about watchdog issues, see [Hardware reset](https://docs.qualcomm.com/doc/80-70022-12/topic/general_system_debugging.html#reset-hardware).
For more information about the debugging issues in Qualcomm TEE, see [Qualcomm Linux Security Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-11/debug.html).
## Debug methods
The following are the two debugging approaches:
- **On-target debugging**
On-target debugging resolves software issues. This approach allows access to most information directly from the live device. Connect the Linux host to the live device using SSH. To set up SSH, see [Sign in using SSH](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-254/how_to.html#use-ssh).
For more information about various on-target debug features, see [On-target debug features](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#on-target-debug-features).
- **Off-target debugging**
This approach uses logs instead of an actual device, making it an efficient method of debugging. You can debug using memory dumps or logging tools and various types of logs for offline debugging. As the RAM dump captures most of the memory regions, but only limited hardware register information, debugging a hardware-related issue can be a challenge.
For more information about various off-target debug features, see [Off-target debug features](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#off-target-debug-features).
## On-target debug features
You can use the following debug features on the device at runtime:
- [Perf utility](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#section-tmb-3v5-41c)
- [Subsystem restart](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#section-o5w-vfb-pzb)
- [Force subsystem reset](https://docs.qualcomm.com/doc/80-70022-12/topic/Debug-overview.html#section-gdz-gjb-pzb)
- [Debug using OpenOCD](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#debug-using-openocd)
### Perf utility
The Perf utility in Linux facilitates performance analysis and profiling. The Linux kernel includes this utility at the `tools/perf` directory.
You can use the Perf utility to debug various aspects of system behavior, including CPU performance counters, tracepoints, kprobes, and uprobes (for dynamic tracing). The following table lists the key features of the Perf utility.
Table : Key features of Perf utility
| Feature | Description |
| --- | --- |
| CPU performance counters | These CPU hardware registers track events such as executed instructions, cache misses, and branch mispredictions. These events form the basis for profiling applications and identifying performance bottlenecks. |
| Tracepoints | You can place these tracepoints at logical locations in the code, such as system calls, network events, and file system operations. These tracepoints provide information such as timestamps and stack traces with minimal overhead. |
| Dynamic tracing | This feature dynamically creates tracepoints using the kprobes and uprobes frameworks, allowing tracing in both kernel space and user space. |
For more information about the Perf utility, see the following resources:
- [https://perf.wiki.kernel.org/index.php/Main_Page](https://perf.wiki.kernel.org/index.php/Main_Page)
- [https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md](https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md)
- [https://docs.kernel.org/trace/coresight/index.html](https://docs.kernel.org/trace/coresight/index.html)
- [https://docs.kernel.org/trace/coresight/coresight-perf.html](https://docs.kernel.org/trace/coresight/coresight-perf.html)
### Subsystem restart
Subsystems can restart independently when they crash without requiring a device restart, called a subsystem restart (SSR). The SSR feature uses the remoteproc framework available in the Linux kernel.
To enable a full memory dump for debugging purposes, disable the SSR feature by default. It's recommended to use the SSR feature on commercial devices and not during the development phase. For more information about the SSR feature, see [Configure
the remoteprocessor subsystems](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-3/remoteproc-overview.html).
Enable the SSR feature to generate the SSR dump on a subsystem restart. Consider using the SSR dump for debugging, as its size is smaller than the full memory dump. However, you need the entire RAM dump to debug some subsystem crashes.
For more information about how to enable and capture subsystem dumps, see [Capture the subsystem dumps](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#subsystem-ram-dumps).
### Force subsystem reset
Use this feature to debug use cases that require restarting a subsystem.
This feature is available to licensed developers with authorized access. To forcibly reset the subsystem, you can run the `diag` command using the QXDM Professional^™^. For more information about QXDM Professional commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70022-12A/debug-non-hlos-subsystems.html#qxdm_professional).
## Off-target debug features
You can use RAM dump or QXDM Professional for offline debugging.
### RAM dump
The RAM dump is a snapshot of the entire memory at the time of failure. You can analyze the RAM dump using various tools, including Qualcomm^®^ Crash Analysis Portal (QCAP), RAMParser, Lauterbach TRACE32 simulator, and crash utility.
For more information about how to collect and parse RAM dumps, see [Collect and parse RAM dump](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#collect-and-parse-ram-dumps).
### QXDM Professional
Use QXDM Professional to debug various subsystems.
This tool is available to licensed developers with authorized access. For more information about QXDM Professional commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70022-12A/debug-non-hlos-subsystems.html#qxdm_professional).
## Common system issues
You can categorize system issues into one of the following types:
>
>
> - [Linux kernel space issues](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#debugging-linux-kernel)
> - [System issues](https://docs.qualcomm.com/doc/80-70022-12/topic/general_system_debugging.html#general-system-debugging)
>
>
>
> >
> >
> > - [Watchdog issues](https://docs.qualcomm.com/doc/80-70022-12/topic/general_system_debugging.html#watchdog-timeout)
> > - [Bus hang and timeout issues](https://docs.qualcomm.com/doc/80-70022-12/topic/general_system_debugging.html#erroneous-transaction-on-bus-error-and-timeout)
> > - [Hardware reset issues](https://docs.qualcomm.com/doc/80-70022-12/topic/general_system_debugging.html#reset-hardware)
## Miscellaneous issues
You may run into the following miscellaneous issues:
- Device-freeze issues due to software bugs
- Linux application-related issues
- Random resets due to hardware issues related to the PCB
## Next steps
>
>
> - To debug Linux application-related issues, see [Debug Linux user space issues](https://docs.qualcomm.com/doc/80-70022-12/topic/using_open_source_debug_tools.html#using-open-source-debug-tools).
> - To debug Linux kernel-related issues, see [Debug Linux kernel space issues](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#debugging-linux-kernel).
Last Published: Oct 15, 2025
[Previous Topic
Debug documentation](https://docs.qualcomm.com/bundle/publicresource/80-70022-12/topics/landing_page.md) [Next Topic
Debug Linux user space issues](https://docs.qualcomm.com/bundle/publicresource/80-70022-12/topics/using_open_source_debug_tools.md)