Page-1

# Debug overview A subsystem processes independently within its own execution environment on the Qualcomm^®^ SoC. This section describes the tools, sample logs, and troubleshooting methods for impacted subsystems. Understanding the discrepancies in subsystems and methods for examining errors is useful for diagnosing and troubleshooting such anomalies. The following table lists the important subsystems on the SoC. Table: Subsystems on SoC | Subsystem | Description | | --- | --- | | Application processor subsystem (APSS) | This primary subsystem executes the Qualcomm Linux kernel as the high-level OS (HLOS). | | Application digital signal processor (aDSP) | This subsystem, known as the low-power audio subsystem (LPASS), handles digital signal processing tasks, such as audio encoding/decoding, and voice recognition. | | Compute digital signal processor (cDSP) | This subsystem, known as the compute DSP, performs compute-intensive tasks, such as neural network-related calculations. | | Qualcomm^®^ Trusted Execution Environment (TEE)/TrustZone (TZ) | This subsystem performs secure operations and leverages the Arm® TrustZone® architecture. For more information, see [Qualcomm Linux Security Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-11). | | Wireless local area network processor subsystem (WPSS) | This subsystem connects to a Wi-Fi network, transmits and receives data packets, handles security protocols, and ensures stable network performance. For more information, see [Qualcomm Linux Wi-Fi Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-14/introduction.html). | | Always On Processor (AOP) | This subsystem regulates the power on the device. | Software for all subsystems, except the application processor, is known as non-HLOS. To understand the boot flow of the subsystems, see [Qualcomm Linux Boot Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-4). Note See [Hardware SoCs](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-115/soc.html) that Qualcomm^®^ Linux^®^ supports. ## Debug workflow Identify the subsystem where the issue occurred first, as many subsystems are present on the Qualcomm SoC. This helps you debug the relevant subsystem. The application processor, as the primary subsystem detects when other subsystems crash. For example, if the aDSP processor subsystem crashes, the kernel log captures the subsystem restart (SSR) crash error log. Therefore, to identify the subsystem that need debugging, first verify the kernel debug messages. The following figure shows the workflow to identify errors and the components you need to debug. **Figure : Workflow to identify the impacted subsystem** When the kernel dmesg logs show no panic signatures, verify the Qualcomm TEE diagnostic (diag) logs for errors such as nonsecure watchdog bite and network-on-chip (NoC) errors. The following sections describe example kernel messages that indicate kernel panic, bugs, and subsystem crash issues. ### Identify kernel panic and bugs You can determine from the kernel dmesg logs whether the reset was a kernel panic or a bug. The following are some example logs that indicate a kernel panic or a bug: - Panic (Pattern: General error) 18.800936: <6> Kernel panic - not syncing: Fatal exception 18.800938: <6> SMP: stopping secondary CPUs 18.800947: <6> CPU0: stopping Kernel panic - not syncing: Apps watchdog Bark received! - Bug (Pattern 1: Forced crash due to memory corruption) 12.899532: <6> BUG kmalloc-128 (Not tainted): Redzone overwritten 12.905418: <6> ----------------------------------------------------------------------------- - Bug (Pattern 2: Forced crash from driver) 320.510769: <6> ------------[ cut here ]------------ 320.510781: <6> kernel BUG at /local/mnt/workspace/lnxbuild/project/trees\_in\_use/free\_tree\_platform\_manifest\_refs\_tags/drivers/platform/msm/ipa/ipa\_v3/ipa\_qmi\_service.c:955! - Bug (Pattern 3: Forced crash due to lockup issue) [ 180.993861] BUG: spinlock lockup suspected on CPU#0, swapper/0/0 [ 180.993883] lock: stop\_lock+0x0/0x18, .magic: dead4ead, .owner: swapper/6/0, .owner\_cpu: 6 [ 181.015629] Causing a watchdog bite! Note Most bugs typically result in a kernel panic. For more information about debugging kernel issues, see [Qualcomm Linux Kernel Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-3/overview.html). ### Identify subsystem crash issues You can verify the kernel dmesg logs to determine whether the restart was due to a subsystem crash. For example, the following log indicates that the aDSP subsystem crashed, according to the `qcm6490.dtsi` (node: remoteproc\_adsp: remoteproc@3000000). 0x000000000A27652C | 5198.790423: qcom\_q6v5\_pas 3000000.remoteproc: fatal error received: err\_inject\_crash.c:413:Crash injected via Diag 0x000000000A276689 | 5198.801061: remoteproc remoteproc2: crash detected in 3000000.remoteproc: type fatal error 0x000000000A2767A1 | 5198.809602: remoteproc remoteproc2: handling crash #1 in 3000000.remoteproc 0x000000000A27688E | 5198.816837: remoteproc remoteproc2: recovering 3000000.remoteproc 0x000000000A276971 | 5198.823784: qcom\_q6v5\_pas 8a00000.remoteproc: subsystem event rejected ### Identify system issues from Qualcomm TEE logs This feature is available to licensed developers with authorized access. For sample logs that indicate errors in the Qualcomm TEE, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70018-12A/identify_subsystem_to_be_debugged.html#identify_and_debug_errors_in_tz). If the Qualcomm TEE diag logs shows no error, the issue may be due to a secure watchdog bite issue. For more information about watchdog issues, see [Hardware reset](https://docs.qualcomm.com/doc/80-70018-12/topic/general_system_debugging.html#reset-hardware). For more information about the debugging issues in Qualcomm TEE, see [Qualcomm Linux Security Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-11/debug.html). ## Debug methods The following are the two debugging approaches: - **On-target debugging** On-target debugging resolves software issues. This approach allows access to most information directly from the live device. Connect the Linux host to the live device using SSH. To set up SSH, see [Sign in using SSH](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-254/how_to.html#use-ssh). For more information about various on-target debug features, see [On-target debug features](https://docs.qualcomm.com/doc/80-70018-12/topic/Debug-overview.html#on-target-debug-features). - **Off-target debugging** This approach uses logs instead of an actual device, making it an efficient method of debugging. You can debug using memory dumps or logging tools and various types of logs for offline debugging. As the RAM dump captures most of the memory regions, but only limited hardware register information, debugging a hardware-related issue can be a challenge. For more information about various off-target debug features, see [Off-target debug features](https://docs.qualcomm.com/doc/80-70018-12/topic/Debug-overview.html#off-target-debug-features). ## On-target debug features You can use the following debug features on the device at runtime: - Perf utility - Subsystem restart - Force subsystem reset - [Debug using OpenOCD](https://docs.qualcomm.com/doc/80-70018-12/topic/debugging_linux_kernel.html#debug-using-openocd) ### Perf utility The Perf utility in Linux facilitates performance analysis and profiling. The Linux kernel includes this utility at the `tools/perf` directory. You can use the Perf utility to debug various aspects of system behavior, including CPU performance counters, tracepoints, kprobes, and uprobes (for dynamic tracing). The following table lists the key features of the Perf utility. Table : Key features of Perf utility | Feature | Description | | --- | --- | | CPU performance counters | These CPU hardware registers track events such as executed instructions, cache misses, and branch mispredictions. These events form the basis for profiling applications and identifying performance bottlenecks. | | Tracepoints | You can place these tracepoints at logical locations in the code, such as system calls, network events, and file system operations. These tracepoints provide information
such as timestamps and stack traces with minimal overhead. | | Dynamic tracing | This feature dynamically creates tracepoints using the kprobes and uprobes frameworks, allowing tracing in both kernel space and user space. | For more information about the Perf utility, see the following resources: - [https://perf.wiki.kernel.org/index.php/Main_Page](https://perf.wiki.kernel.org/index.php/Main_Page) - [https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md](https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md) - [https://docs.kernel.org/trace/coresight/index.html](https://docs.kernel.org/trace/coresight/index.html) - [https://docs.kernel.org/trace/coresight/coresight-perf.html](https://docs.kernel.org/trace/coresight/coresight-perf.html) ### Subsystem restart Subsystems can restart independently when they crash without requiring a device restart, called a subsystem restart (SSR). The SSR feature uses the remoteproc framework available in the Linux kernel. To enable a full memory dump for debugging purposes, disable the SSR feature by default. It's recommended to use the SSR feature on commercial devices and not during the development phase. For more information about the SSR feature, see [Remoteproc subsystem](https://docs.qualcomm.com/bundle/publicresource/topics/80-70018-3/features.html#remoteproc-subsystem). Enable the SSR feature to generate the SSR dump on a subsystem restart. Consider using the SSR dump for debugging, as its size is smaller than the full memory dump. However, you need the entire RAM dump to debug some subsystem crashes. For more information about how to enable and capture subsystem dumps, see [Subsystem dumps](https://docs.qualcomm.com/doc/80-70018-12/topic/debugging_linux_kernel.html#subsystem-ram-dumps). ### Force subsystem reset Use this feature to debug use cases that require restarting a subsystem. This feature is available to licensed developers with authorized access. To forcibly reset the subsystem, you can run the `diag` command using the QXDM Professional^™^. For more information about QXDM Professional commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70018-12A/debug-non-hlos-subsystems.html#qxdm_professional). ## Off-target debug features You can use RAM dump or QXDM Professional for offline debugging. ### RAM dump The RAM dump is a snapshot of the entire memory at the time of failure. You can analyze the RAM dump using various tools, including Qualcomm^®^ Crash Analysis Portal (QCAP), RAMParser, Lauterbach TRACE32 simulator, and crash utility. For more information about how to collect and parse RAM dumps, see [Collect and parse RAM dump](https://docs.qualcomm.com/doc/80-70018-12/topic/debugging_linux_kernel.html#collect-and-parse-ram-dumps). ### QXDM Professional Use QXDM Professional to debug various subsystems. This tool is available to licensed developers with authorized access. For more information about QXDM Professional commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70018-12A/debug-non-hlos-subsystems.html#qxdm_professional). ## Common system issues You can categorize system issues into one of the following types: > > > - [Linux kernel space issues](https://docs.qualcomm.com/doc/80-70018-12/topic/debugging_linux_kernel.html#debugging-linux-kernel) > - [System issues](https://docs.qualcomm.com/doc/80-70018-12/topic/general_system_debugging.html#general-system-debugging) > > > > > > > > > - [Watchdog issues](https://docs.qualcomm.com/doc/80-70018-12/topic/general_system_debugging.html#watchdog-timeout) > > - [Bus hang and timeout issues](https://docs.qualcomm.com/doc/80-70018-12/topic/general_system_debugging.html#erroneous-transaction-on-bus-error-and-timeout) > > - [Hardware reset issues](https://docs.qualcomm.com/doc/80-70018-12/topic/general_system_debugging.html#reset-hardware) ## Miscellaneous issues You may run into the following miscellaneous issues: - Device-freeze issues due to software bugs - Linux application-related issues - Random resets due to hardware issues related to the PCB To debug Linux kernel-related issues, see [Debug Linux kernel space issues](https://docs.qualcomm.com/doc/80-70018-12/topic/debugging_linux_kernel.html#debugging-linux-kernel). To debug Linux application-related issues, see [Debug Linux user space issues](https://docs.qualcomm.com/doc/80-70018-12/topic/using_open_source_debug_tools.html#using-open-source-debug-tools). Last Published: May 02, 2025 [Next Topic Debug Linux user space issues](https://docs.qualcomm.com/bundle/publicresource/80-70018-12/topics/using_open_source_debug_tools.md)