Page-1

# Overview A subsystem is a processor that has its own independent execution environment on the Qualcomm^®^ SoC. The tools, sample logs, and troubleshooting methods for the impacted subsystems are described here. Understanding the discrepancies that occur in subsystems and the methods for examining errors is useful when you must diagnose and troubleshoot such anomalies. The following table lists the important subsystems on the SoC. Table : Subsystems on SoC | Subsystem | Description | | --- | --- | | Application processor subsystem (APSS) | This is the primary subsystem. It executes the Linux kernel as the high-level OS (HLOS). | | Application digital signal processor (aDSP) | This subsystem is also known as the low-power audio subsystem (LPASS). This subsystem handles digital signal processing tasks like audio encoding/decoding, and voice recognition. | | Compute digital signal processor (cDSP) | This subsystem is also known as the compute DSP. This subsystem is intended for compute-intensive tasks such as neural network-related calculations. | | Qualcomm^®^ Trusted Execution Environment (TEE)/TrustZone (TZ) | This subsystem performs secure operations and is built on the Arm® TrustZone® architecture. For more information on this subsystem, see [Qualcomm Linux Security Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-11). | | Wireless local area network processor subsystem (WPSS) | This subsystem manages the task of connecting to a Wi-Fi network, transmitting and receiving data packets, handling security protocols, and ensuring stable network performance. For more information on this subsystem, see [Qualcomm Linux Wi-Fi Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-14/introduction.html). | | Always On Processor (AOP) | This subsystem regulates the power on the device. | Except for the application processor, software for all other subsystems is also known as non-HLOS. To understand the boot flow of the subsystems, see [Qualcomm Linux Boot Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-4). Note See [Hardware SoCs](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-115/soc.html) that are supported on Qualcomm^®^ Linux^®^. ## Debug workflow Because multiple subsystems are present on the Qualcomm SoC, it is important to first identify the subsystem on which the issue occurred so that you can debug the relevant subsystem. The application processor is the primary subsystem, and it can detect when other subsystems crash. For example, if the aDSP processor subsystem crashes, the kernel log captures the subsystem restart (SSR) crash error log. Therefore, to identify the subsystems to be debugged, you must first verify the kernel debug messages. The following figure shows the workflow to identify errors and the components to be debugged. **Figure : Workflow to identify the impacted subsystem** When there are no panic signatures in the kernel dmesg logs, verify the Qualcomm TEE diagnostic (diag) logs for errors such as nonsecure watchdog bite, and network-on-chip (NoC) errors. The following topics describe some example kernel messages that indicate kernel panic, bugs, and subsystem crash issues. ### Identify kernel panic and bugs From the kernel dmesg logs, you can determine whether the reset was a kernel panic or a bug. The following are some example logs that indicate a kernel panic or a bug: - Panic (Pattern: General error) 18.800936: <6> Kernel panic - not syncing: Fatal exception 18.800938: <6> SMP: stopping secondary CPUs 18.800947: <6> CPU0: stopping Kernel panic - not syncing: Apps watchdog Bark received! - Bug (Pattern 1: Forced crash due to memory corruption) 12.899532: <6> BUG kmalloc-128 (Not tainted): Redzone overwritten 12.905418: <6> ----------------------------------------------------------------------------- - Bug (Pattern 2: Forced crash from driver) 320.510769: <6> ------------[ cut here ]------------ 320.510781: <6> kernel BUG at /local/mnt/workspace/lnxbuild/project/trees\_in\_use/free\_tree\_platform\_manifest\_refs\_tags/drivers/platform/msm/ipa/ipa\_v3/ipa\_qmi\_service.c:955! - Bug (Pattern 3: Forced crash due to lockup issue) [ 180.993861] BUG: spinlock lockup suspected on CPU#0, swapper/0/0 [ 180.993883] lock: stop\_lock+0x0/0x18, .magic: dead4ead, .owner: swapper/6/0, .owner\_cpu: 6 [ 181.015629] Causing a watchdog bite! Note Most bugs are followed by a kernel panic. For more information on debugging kernel issues, see [Qualcomm Linux Kernel Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-3/overview.html). ### Identify subsystem crash issues To determine whether the restart was due to a subsystem crash, verify the kernel dmesg logs. For example, the following log indicates that the aDSP subsystem crashed, according to the `qcm6490.dtsi` (node: remoteproc\_adsp: remoteproc@3000000). 0x000000000A27652C | 5198.790423: qcom\_q6v5\_pas 3000000.remoteproc: fatal error received: err\_inject\_crash.c:413:Crash injected via Diag 0x000000000A276689 | 5198.801061: remoteproc remoteproc2: crash detected in 3000000.remoteproc: type fatal error 0x000000000A2767A1 | 5198.809602: remoteproc remoteproc2: handling crash #1 in 3000000.remoteproc 0x000000000A27688E | 5198.816837: remoteproc remoteproc2: recovering 3000000.remoteproc 0x000000000A276971 | 5198.823784: qcom\_q6v5\_pas 8a00000.remoteproc: subsystem event rejected ### Identify system issues from Qualcomm TEE logs This feature is available to licensed developers with authorized access. For sample logs that indicate errors in the Qualcomm TEE, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70017-12A/identify_subsystem_to_be_debugged.html#identify_and_debug_errors_in_tz). If there is no error in the Qualcomm TEE diag logs, it is most probably a secure watchdog bite issue. For more information on watchdog issues, see [Hardware reset](https://docs.qualcomm.com/doc/80-70017-12/topic/general_system_debugging.html#reset-hardware). For information on debugging issues in Qualcomm TEE, see [Qualcomm Linux Security Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-11/debug.html). ## Debugging methods There are two approaches to debugging: - **On-target debugging** On-target debugging is a basic way of debugging software issues. This approach is powerful because most of the information is accessible from the live device. SSH is used to connect the Linux host to the live device. To set up SSH, see [Use SSH](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-254/how_to.html#use-ssh). For more information about various on-target debug features, see [On-target debug features](https://docs.qualcomm.com/doc/80-70017-12/topic/Debug-overview.html#on-target-debug-features). - **Off-target debugging** This debugging approach requires logs instead of an actual device, and is a convenient and efficient method of debugging. Debugging is performed using memory dumps or logging tools. Various types of logs can be used for offline debugging. Because the RAM dump captures most of the memory regions, but only limited hardware register information, debugging a hardware-related issue can be a challenge. For more information about various off-target debug features, see [Off-target debug features](https://docs.qualcomm.com/doc/80-70017-12/topic/Debug-overview.html#off-target-debug-features). ## On-target debug features The following debug features are available on the device and can be used at runtime: - Perf utility - Subsystem restart - Force subsystem reset ### Perf utility Perf is a powerful utility in Linux that facilitates performance analysis and profiling. This utility is included in the Linux kernel at `tools/perf` . The Perf utility can be used to debug various aspects of system behavior, including CPU performance counters, tracepoints, kprobes, and uprobes (for dynamic tracing). The following table lists the key features of the Perf utility. Table : Key features of Perf utility | Feature | Description | | --- | --- | | CPU performance counters | These CPU hardware registers are used to track events such as executed instructions, cache misses, and branch mispredictions. These events form the basis for profiling applications and identifying performance bottlenecks. | | Tracepoints | These tracepoints are placed at logical locations in the code. For example, system calls, network events, and file system operations. These tracepoints provide information such as
timestamps and stack traces with minimal overhead. | | Dynamic tracing | This feature dynamically creates tracepoints using the kprobes and uprobes frameworks. Therefore, this feature allows tracing in both kernel space and user space. | For more information on the Perf utility, see the following resources: - [https://perf.wiki.kernel.org/index.php/Main_Page](https://perf.wiki.kernel.org/index.php/Main_Page) - [https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md](https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md) - [https://docs.kernel.org/trace/coresight/index.html](https://docs.kernel.org/trace/coresight/index.html) - [https://docs.kernel.org/trace/coresight/coresight-perf.html](https://docs.kernel.org/trace/coresight/coresight-perf.html) ### Subsystem restart The subsystems can restart independently when they crash without requiring a reboot of the device. This type of restart is known as a subsystem restart (SSR). The SSR feature is implemented using the remoteproc framework available in the Linux kernel. To enable a full memory dump for debugging purposes, the SSR feature is disabled by default. It is recommended to use the SSR feature on commercial devices and not during the development phase. For more information on the SSR feature, see [Remoteproc subsystem](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-3/features.html#remoteproc-subsystem). When the SSR feature is enabled, the SSR dump is generated on a subsystem restart. You may consider using the SSR dump for debugging, as its size is smaller than the full memory dump. However, the entire RAM dump is required to debug some subsystem crashes. For information on how to enable and capture subsystem dumps, see [Subsystem dumps](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#subsystem-ram-dumps). ### Force subsystem reset This feature can be used to debug use cases that require you to restart a subsystem. This feature is available to licensed developers with authorized access. To reset the subsystem forcibly, you can run the `diag` command using the QXDM Professional^™^. For information on QXDM Professional commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70017-12A/debug-non-hlos-subsystems.html#qxdm_professional). ## Off-target debug features You can use RAM dump or QXDM Professional for offline debugging. ### RAM dump The RAM dump is a snapshot of the entire memory at the time of failure. The RAM dump can be analyzed using various tools, including Qualcomm Crash Analysis Portal (QCAP), RAMParser, Lauterbach TRACE32 simulator, and crash utility. For information on how to collect and parse RAM dumps, see [Collect and parse RAM dump](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#collect-and-parse-ram-dumps). ### QXDM Professional QXDM Professional is used to debug various subsystems. This tool is available to licensed developers with authorized access. For information on QXDM Professional commands and their usage, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70017-12A/debug-non-hlos-subsystems.html#qxdm_professional). ## Common system issues System issues can be categorized into one of the following types: > > > - [Linux kernel space issues](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#debugging-linux-kernel) > - [System issues](https://docs.qualcomm.com/doc/80-70017-12/topic/general_system_debugging.html#general-system-debugging) > > > > > > > > > - [Watchdog issues](https://docs.qualcomm.com/doc/80-70017-12/topic/general_system_debugging.html#watchdog-timeout) > > - [Bus hang and timeout issues](https://docs.qualcomm.com/doc/80-70017-12/topic/general_system_debugging.html#erroneous-transaction-on-bus-error-and-timeout) > > - [Hardware reset issues](https://docs.qualcomm.com/doc/80-70017-12/topic/general_system_debugging.html#reset-hardware) ## Miscellaneous issues The following are the miscellaneous issues that you may encounter: - Device-freeze issues due to software bugs - Linux application-related issues - Random resets due to hardware issues related to the PCB To debug Linux kernel-related issues, see [Debug Linux kernel space issues](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#debugging-linux-kernel). To debug Linux application-related issues, see [Debug Linux user space issues](https://docs.qualcomm.com/doc/80-70017-12/topic/using_open_source_debug_tools.html#using-open-source-debug-tools). Last Published: Dec 25, 2024 [Next Topic Debug Linux user space issues](https://docs.qualcomm.com/bundle/publicresource/80-70017-12/topics/using_open_source_debug_tools.md)