# Overview Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html) A subsystem is a processor that has its own independent execution environment on the Qualcomm^®^ SoC. The tools, sample logs, and troubleshooting methods for the impacted subsystems are described here. Understanding the discrepancies that occur in subsystems and the methods for examining errors is useful when you must diagnose and troubleshoot such anomalies. The following are the important subsystems on the SoC: Table : Subsystems on SoC | Subsystem | Description | | --- | --- | | Application processor subsystem (APSS) | This is the primary subsystem. It executes the Linux kernel as
the high-level OS (HLOS). | | Application digital signal processor (aDSP) | This subsystem is also known as the low-power audio subsystem
(LPASS). | | Compute digital signal processor (cDSP) | This subsystem is also known as the compute DSP. | | Qualcomm^®^ Trusted Execution Environment
(TEE)/TrustZone (TZ) | For more information on this subsystem, see [Qualcomm Linux Security
Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-11). | | Wireless local area network processor subsystem (WPSS) | For more information on this subsystem, see [Qualcomm Linux Wi-Fi
Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-14/introduction.html). | | Always On Processor (AOP) | This subsystem regulates the power on the device. | Except for the application processor, software pertaining to all other subsystems is also known as non-HLOS. To understand the boot flow of the subsystems, see [Qualcomm Linux Boot Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-4). Note: See [Hardware SoCs](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-115/soc.html) that are supported on Qualcomm^®^ Linux^®^. ## Debug workflow Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html) Because multiple subsystems are present on the Qualcomm SoC, it is important to first identify the subsystem on which the issue occurred so that you can debug the relevant subsystem. The application processor is the primary subsystem, and it can detect when other subsystem crashes occur. For example, if the aDSP processor subsystem crashes, the kernel log captures the subsystem restart (SSR) crash error log. Therefore, to identify the subsystems to be debugged, you must first verify the kernel debug messages. The following figure shows the workflow to identify errors and the components to be debugged. Figure : Workflow to identify the impacted subsystem Page-1 Start Start Panic or Bug in dmesg? Panic or Bugin dmesg? Subsystem restart log in dmesg? Subsystem restart log in dmesg? Error in TrustZone? Error in TrustZone? Yes Yes No No Debug specific subsystem Debug specific subsystem Debug Application processor Debug Application processor Debug Always-on processor Debug Always-on processor Debug NoC Debug NoC Yes Yes No No Nonsecure WD bite AHB timeout XPU error SMMU error No error Nonsecure WD biteAHB timeoutXPU errorSMMU errorNo error NoC error NoC error AoP error AoP error When there are no panic signatures in the kernel dmesg logs, verify the Qualcomm TEE diagnostic (diag) logs for errors such as nonsecure watchdog bite, and network-on-chip (NoC) errors. The following topics describe some of the example kernel messages that indicate kernel panic, bugs, and subsystem crash issues. ### Identify kernel panic and bugs From the kernel dmesg logs, you can determine whether the reset was a kernel panic or a bug. The following are some example logs that indicate a kernel panic or a bug: - Panic (Pattern: General error) 18.800936: <6> Kernel panic - not syncing: Fatal exception 18.800938: <6> SMP: stopping secondary CPUs 18.800947: <6> CPU0: stopping Kernel panic - not syncing: Apps watchdog Bark received!Copy to clipboard - Bug (Pattern 1: Forced crash due to memory corruption) 12.899532: <6> BUG kmalloc-128 (Not tainted): Redzone overwritten 12.905418: <6> -----------------------------------------------------------------------------Copy to clipboard - Bug (Pattern 2: Forced crash from driver) 320.510769: <6> ------------[ cut here ]------------ 320.510781: <6> kernel BUG at /local/mnt/workspace/lnxbuild/project/trees_in_use/free_tree_platform_manifest_refs_tags/drivers/platform/msm/ipa/ipa_v3/ipa_qmi_service.c:955!Copy to clipboard - Bug (Pattern 3: Forced crash due to lockup issue) [ 180.993861] BUG: spinlock lockup suspected on CPU#0, swapper/0/0 [ 180.993883] lock: stop_lock+0x0/0x18, .magic: dead4ead, .owner: swapper/6/0, .owner_cpu: 6 [ 181.015629] Causing a watchdog bite!Copy to clipboard Note: Most bugs are followed by a kernel panic. For more information on debugging kernel issues, see [Qualcomm Linux Kernel Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-3/overview.html). ### Identify subsystem crash issues To determine whether the restart was due to a subsystem crash, verify the kernel dmesg logs. For example, the following log indicates that the aDSP subsystem crashed, according to the qcm6490.dtsi (node: remoteproc\_adsp: remoteproc@3000000). 0x000000000A27652C | 5198.790423: qcom_q6v5_pas 3000000.remoteproc: fatal error received: err_inject_crash.c:413:Crash injected via Diag 0x000000000A276689 | 5198.801061: remoteproc remoteproc2: crash detected in 3000000.remoteproc: type fatal error 0x000000000A2767A1 | 5198.809602: remoteproc remoteproc2: handling crash #1 in 3000000.remoteproc 0x000000000A27688E | 5198.816837: remoteproc remoteproc2: recovering 3000000.remoteproc 0x000000000A276971 | 5198.823784: qcom_q6v5_pas 8a00000.remoteproc: subsystem event rejectedCopy to clipboard ### Identify system issues from Qualcomm TEE logs This feature is available to licensed developers with authorized access. For sample logs that indicate errors in the Qualcomm TEE, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70015-12A/identify_subsystem_to_be_debugged.html#identify_and_debug_errors_in_tz). If there is no error in the Qualcomm TEE diag logs, it is most probably a secure watchdog bite issue. For more information on watchdog issues, see [Hardware reset](https://docs.qualcomm.com/doc/80-70015-12/topic/general_system_debugging.html#reset_hardware). For information on debugging issues in Qualcomm TEE, see [Qualcomm Linux Security Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-11/debug.html). ## Debugging methods Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html) There are two approaches to debugging: - **On-target debugging** On-target debugging is a basic way of debugging software issues. This approach is powerful because most of the information is accessible from the live device. SSH is used to connect the host machine to the live device. To set up SSH, follow the steps mentioned [here](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-254/how_to.html#use-ssh). For more information about various on-target debug features, see [On-target debug features](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-12/Debug-overview.html#on_target_debug_features). - **Off-target debugging** This debugging approach requires logs instead of an actual device, and is a convenient and efficient method of debugging. Debugging is performed using memory dumps or logging tools. Various types of logs can be used for offline debugging. Because the RAM dump captures most of the memory regions, but only limited hardware register information, debugging a hardware-related issue can be a challenge. For more information about various on-target debug features, see [Off-target debug features](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-12/Debug-overview.html#off_target_debug_features). ## On-target debug features Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html) The following debug features are available on the device and can be used at runtime: ### Perf utility Perf is a powerful utility in Linux that facilitates performance analysis and profiling. This utility is included in the Linux kernel at the tools/perf directory. The Perf utility can be used to debug various aspects of system behavior, including CPU performance counters, tracepoints, kprobes, and uprobes (for dynamic tracing). The following are the key features of the Perf utility: Table : Key features of Perf utility | Feature | Description | | --- | --- | | CPU performance counters | These CPU hardware registers are used to track events such as
executed instructions, cache misses, and branch mispredictions.
These events form the basis for profiling applications and
identifying performance bottlenecks. | | Tracepoints | These tracepoints are placed at logical locations in the
code. For example, system calls, network events, and file system
operations. These tracepoints provide information such as
timestamps and stack traces with minimal overhead. | | Dynamic tracing | This feature dynamically creates tracepoints using the
kprobes and uprobes frameworks. Therefore, this feature allows
tracing in both kernel space and user space. | For more information on the Perf utility, see the following: - [https://perf.wiki.kernel.org/index.php/Main_Page](https://perf.wiki.kernel.org/index.php/Main_Page) - [https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md](https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md) - [https://docs.kernel.org/trace/coresight/index.html](https://docs.kernel.org/trace/coresight/index.html) - [https://docs.kernel.org/trace/coresight/coresight-perf.html](https://docs.kernel.org/trace/coresight/coresight-perf.html) ### Subsystem restart The subsystems can restart independently when they crash without requiring a reboot of the device. This type of restart is known as a subsystem restart (SSR). The SSR feature is implemented using the remoteproc framework available in the Linux kernel. To enable a full memory dump for debugging purposes, the SSR feature is disabled by default. It is recommended to use the SSR feature on commercial devices and not during the development phase. For more information on the SSR feature, see [Remoteproc subsystem](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-3/features.html#remoteproc-subsystem). When the SSR feature is enabled, the SSR dump is generated on a subsystem restart. You may consider using the SSR dump for debugging, as its size is smaller than the full memory dump. However, the entire RAM dump is required to debug some subsystem crashes. For information on how to enable and capture subsystem dumps, see [Subsystem dumps](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html#subsystem_ram_dumps). ### Force subsystem reset This feature can be used to debug use cases that require you to restart a subsystem. This feature is available to licensed developers with authorized access. To reset the subsystem forcibly, you can run the `diag` command using the QXDM Professional™. For information on QXDM Professional commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70015-12A/debug-non-hlos-subsystems.html#qxdm_professional). ## Off-target debug features Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html) The following features and tools are available for offline debugging: ### RAM dump The RAM dump is a snapshot of the entire memory at the time of failure. The RAM dump can be analyzed using various tools, including Qualcomm Crash Analysis Portal (QCAP), RAMParser, Lauterbach TRACE32 simulator, and crash utility. For information on how to collect and parse RAM dumps, see [Collect and parse RAM dump](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html#collect_and_parse_ram_dumps). ### QXDM Professional QXDM Professional can be used to debug various subsystems. This tool is available to licensed developers with authorized access. For information on QXDM Professional commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70015-12A/debug-non-hlos-subsystems.html#qxdm_professional). ## Common system issues Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html) System issues can be categorized into one of the following types: - [Linux kernel space issues](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html) - [System issues](https://docs.qualcomm.com/doc/80-70015-12/topic/general_system_debugging.html) - [Watchdog issues](https://docs.qualcomm.com/doc/80-70015-12/topic/general_system_debugging.html#watchdog_timeout) - [Bus hang and timeout issues](https://docs.qualcomm.com/doc/80-70015-12/topic/general_system_debugging.html#erroneous_transaction_on_bus_error_and_timeout) - [Hardware reset issues](https://docs.qualcomm.com/doc/80-70015-12/topic/general_system_debugging.html#reset_hardware) ## Miscellaneous issues Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70015-12/topic/Debug-overview.html) The following are the miscellaneous issues that you may encounter: - Device-freeze issues due to software bugs - Linux application-related issues - Random resets due to hardware issues related to the PCB To debug Linux kernel-related issues, see [Debug Linux kernel space issues](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html). To debug Linux application-related issues, see [Debug Linux user space issues](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html). Last Published: Oct 14, 2024 [Next Topic Debug Linux user space issues](https://docs.qualcomm.com/bundle/publicresource/80-70015-12/topics/using_open_source_debug_tools.md)