Page-1

# Overview Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html) A subsystem is a processor that has its own independent execution environment on the Qualcomm^®^ Linux device (SoC). The tools, sample logs, and procedures to troubleshoot the affected subsystems are described here. Following are the different subsystems on Qualcomm SoC. Table : Subsystems on SoC | Subsystem | Description | | --- | --- | | Application processor subsystem (APSS) | This is the master subsystem. It executes the Linux kernel as the
high-level OS (HLOS). | | aDSP/LPASS | Application digital signal processor (aDSP) is also known as the
Low-power audio subsystem (LPASS). | | mDSP/MPSS | Modem processor subsystem | | cDSP | Compute DSP | | WPSS | WLAN processor subsystem | | TrustZone (TZ) | This is the Qualcomm Trusted Execution Environment. For more
information on the TrustZone subsystem, see [Qualcomm Linux Security
Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-11). | | Always On Processor (AOP) | This subsystem regulates the power on the device. | Except the application processor, software pertaining to all other subsystems are also known as non-HLOS. To understand the boot flow of the subsystems, see [Qualcomm Linux Boot Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-4). To know more about the chipset specifications and for the functional block diagram, see [QCS6490/QCS5430 Data Sheet](https://docs.qualcomm.com/bundle/publicresource/topics/80-23889-1/device-description.html). For definitions of commonly used terms such as NoC, xPU, see [Acronyms and terms](https://docs.qualcomm.com/doc/80-70014-12/topic/references.html#acronyms_and_terms). Note: The Qualcomm Linux platform allows you to develop applications for QCS6490 and QCS5430. ## Debug workflow Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html) As there are multiple subsystems on the SoC, it is important to first identify the subsystem on which the issue occurred so that you can debug the right subsystem. The application processor is the master processor, and it can detect when the other subsystem crash occurred. For example, if the modem processor subsystem crashes, the kernel log captures the subsystem restart (SSR) crash error log. Therefore, to identify which subsystems have to be debugged, you must first verify the kernel debug messages. The following figure shows the debug workflow to identify the subsystem to be debugged: Figure : Debug workflow to identify the subsystem to be debugged When there are no panic signatures in the kernel dmesg logs, verify the TrustZone Diagostic (diag) logs for errors like Non-Secure watchdog bite, NoC error. Following are some example kernel messages indicating panic, bugs, and subsystem crash issues. ### Identify kernel panic and bugs From the kernel dmesg logs, you can determine if the reset was a kernel panic or bug. Following are some example logs indicating kernel panic or bugs: - Panic (pattern 1: general error) 18.800936: <6> Kernel panic - not syncing: Fatal exception 18.800938: <6> SMP: stopping secondary CPUs 18.800947: <6> CPU0: stopping Kernel panic - not syncing: Apps watchdog Bark received!Copy to clipboard - Bug (pattern 1) 12.899532: <6> BUG kmalloc-128 (Not tainted): Redzone overwritten 12.905418: <6> -----------------------------------------------------------------------------Copy to clipboard - Bug (pattern 2) 320.510769: <6> ------------[ cut here ]------------ 320.510781: <6> kernel BUG at /local/mnt/workspace/lnxbuild/project/trees_in_use/free_tree_platform_manifest_refs_tags/drivers/platform/msm/ipa/ipa_v3/ipa_qmi_service.c:955!Copy to clipboard - Bug (pattern 3) [ 180.993861] BUG: spinlock lockup suspected on CPU#0, swapper/0/0 [ 180.993883] lock: stop_lock+0x0/0x18, .magic: dead4ead, .owner: swapper/6/0, .owner_cpu: 6 [ 181.015629] Causing a watchdog bite!Copy to clipboard Note: Most bugs are followed by a kernel panic. ### Identify subsystem crash issues To determine if the restart was due to a subsystem crash, verify the kernel dmesg logs. For example, the following log indicates that the aDSP subsystem crashed, as per the qcm6490.dtsi (node: remoteproc\_adsp: remoteproc@3000000). 0x000000000A27652C | 5198.790423: qcom_q6v5_pas 3000000.remoteproc: fatal error received: err_inject_crash.c:413:Crash injected via Diag 0x000000000A276689 | 5198.801061: remoteproc remoteproc2: crash detected in 3000000.remoteproc: type fatal error 0x000000000A2767A1 | 5198.809602: remoteproc remoteproc2: handling crash #1 in 3000000.remoteproc 0x000000000A27688E | 5198.816837: remoteproc remoteproc2: recovering 3000000.remoteproc 0x000000000A276971 | 5198.823784: qcom_q6v5_pas 8a00000.remoteproc: subsystem event rejectedCopy to clipboard ### Identify system issues from TrustZone logs This feature is currently available for users who have full access to the proprietary software shipped with Qualcomm Linux. For sample logs indicating errors in the TrustZone, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70014-12A/identify_subsystem_to_be_debugged.html#identify_and_debug_errors_in_tz). If there is no error in the TrustZone diag logs, it is most probably a secure watchdog bite issue. For more information on watchdog issues, see [Hardware reset](https://docs.qualcomm.com/doc/80-70014-12/topic/general_system_debugging.html#reset_hardware). For information on debugging issues in TrustZone, see [Qualcomm Linux Security Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-11/debug.html). ## Debugging features Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html) There are two approaches in debugging: - **On-target debugging** On-target debugging is a basic way of debugging software issues. This approach is powerful as most of the information is accessible from the live device. However, this approach requires that the issue is easily reproducible, and it also requires additional hardware components like the host PC, debugger. SSH is the primary tool to connect host with the device. To use SSH, you must enable SELinux Permissive mode by following the steps mentioned [here](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-254/how_to.html#how-to-ssh-). - **Off-target debugging** This debugging approach requires logs instead of an actual device and is a convenient and efficient method of debugging. Debugging is done using memory dump or logging tools. Various types of logs can be used for offline debugging. You can use log files saved in the embedded file system too. Since the RAM dump captures most of the memory region but limited hardware register information, debugging a hardware-related issue is a challenge. ## On-target debug features Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html) The following debug features are available on the device and can be used at runtime. ### Perf utility Perf is a powerful utility in the Linux ecosystem that facilitates performance analysis and profiling. This utility is included in the Linux kernel at the tools/perf directory. The Perf utility can help debug various aspects of system behavior, including CPU performance counters, tracepoints, kprobes, and uprobes (for dynamic tracing). Following are the key features of the Perf utility: - **CPU performance counters**: These are the CPU hardware registers used to track events like executed instructions, cache misses, and branch mispredictions. These events form the basis for profiling applications and identifying performance bottlenecks. - **Tracepoints**: These are the tracepoints placed at logical locations in the code. For example, system calls, network events, file system operations. These tracepoints provide information like timestamps and stack traces with minimal overhead. - **Dynamic tracing**: Perf can dynamically create tracepoints using the kprobes and uprobes frameworks. Therefore, this allows tracing in both kernel and user space. For more information on the Perf utility, see the following: - [https://docs.kernel.org/trace/coresight/coresight-perf.html](https://docs.kernel.org/trace/coresight/coresight-perf.html) - [https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md](https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md) - [https://docs.kernel.org/trace/coresight/index.html](https://docs.kernel.org/trace/coresight/index.html) - [https://perf.wiki.kernel.org/index.php/Tutorial](https://perf.wiki.kernel.org/index.php/Tutorial) ### Subsystem restart The slave subsystems can restart independently when they crash without requiring a device reboot. This feature of slave restart is known as Subsystem Restart (SSR). It is recommended to use the SSR feature only on commercial devices and not during the development phase. This feature is implemented using the remoteproc framework available in the upstream Linux kernel. By default, the SSR is disabled to enable a full memory dump for debugging issues. For more information on the SSR feature, see [Remoteproc subsystem](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-3/features.html#Toc187817603). The SSR dump can be enabled as part of the SSR operation. The SSR dump is generated when a subsystem restarts and if the SSR dump is enabled. You may prefer using the SSR dump for debugging as the size of the SSR dump is smaller than the full memory dump. However, the entire RAM dump is required to debug certain subsystem crashes. For information on how to enable and capture Subsystem dumps, see [Subsystem dumps](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html#subsystem_ram_dumps). ### Force subsystem reset This feature can be used in debugging use cases that require you to restart a subsystem. This feature is currently available for users who have full access to the proprietary software shipped with Qualcomm Linux. To force reset the subsystem, you can run the diag command using the QXDM tool. For information on QXDM tool commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70014-12A/debug-non-hlos-subsystems.html#qxdm_professional). ## Off-target debug features Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html) Following features and tools are available for offline debugging. ### RAM dump RAM dump is a snapshot of the entire memory (RAM) at the time of failure. RAM dump can be analyzed using various tools including QCAP, RAM parser, and TRACE32 simulator. For information on how to collect and parse RAM dumps, see [Collect and parse RAM dumps](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html#collect_and_parse_ram_dumps). ### QXDM Professional™ QXDM Professional tool can be used to debug various subsystems. This tool is currently available for users who have full access to the proprietary software shipped with Qualcomm Linux. For information on QXDM tool commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70014-12A/debug-non-hlos-subsystems.html#qxdm_professional). ## Common issues Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html) If there is a system malfunction, it can be categorized into one of the following: - [Linux kernel space issues](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html) - [System issues](https://docs.qualcomm.com/doc/80-70014-12/topic/general_system_debugging.html) - [Watchdog issues](https://docs.qualcomm.com/doc/80-70014-12/topic/general_system_debugging.html#watchdog_timeout) - [Bus hang and timeout issues](https://docs.qualcomm.com/doc/80-70014-12/topic/general_system_debugging.html#erroneous_transaction_on_bus_error_and_timeout) - [Hardware reset issues](https://docs.qualcomm.com/doc/80-70014-12/topic/general_system_debugging.html#reset_hardware) ## Miscellaneous issues Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html](https://docs.qualcomm.com/doc/80-70014-12/topic/Debug-overview.html) Following are the miscellaneous issues that you may encounter: - Device freeze issues due to software bugs - Linux application related issues - Random resets due to hardware issues related to PCB To debug Linux kernel related issues, see [Debug Linux kernel space](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html). To debug Linux application related issues, see [Debug Linux user space](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html). Last Published: Jul 12, 2024 [Next Topic Debug Linux user space](https://docs.qualcomm.com/bundle/publicresource/80-70014-12/topics/using_open_source_debug_tools.md)