# Debug Use the following methods to understand the high-level categorization of ways to debug the kernel, and report issues identified with the inherent debug features in the Qualcomm Linux kernel. For more information on debug, see [Qualcomm Linux Debug Guide](bundle/publicresource/topics/80-70015-12). ## Serial console Standard boot and kernel logs are accessible using a serial console. The serial console enables live debugging on a live device. The Qualcomm Linux kernel enables the `CONFIG_SERIAL_QCOM_GENI` driver to support UART and console. Add the following line to the kernel command line parameter: console=ttyMSM0,115200n8 # as in following line in meta-qcom-hwe/conf/machine/include/qcom-.conf KERNEL_CMDLINE_EXTRA ?= "root=/dev/disk/by-partlabel/system rw rootwait console=ttyMSM0,115200n8 pcie_pme=nomsi earlycon" Copy to clipboard When you rebuild and load the kernel: - Connect a Micro USB to USB-A type cable as follows: - Device > Micro USB====USB-TypeA > Host machine - To connect to a device serial port on the host, use a serial console client on a host machine. - You can see serial console logs and other kernel logs on serial client. For more information on the serial console, see [Linux Serial Console](https://docs.kernel.org/admin-guide/serial-console.html). ## Configure console log level Configure the kernel console log level for extensive or minimal logging to satisfy debug or release software versions. Use `/proc/sys/kernel/printk` to configure log level to control log messages appearing on the console. Set the log level from 1 to 7. When you set the log level value to 1, it filters the log to the lowest level. When the log level value is set to 1, only `pr_emerg/KERN_EMERG` printk logs are printed. When you set the log level value to 7, it enables the highest log level `pr_info/KERN_INFO` and prints all printk logs to the console. $ echo "1" > /proc/sys/kernel/printk Copy to clipboard ## Display kernel logs To display the kernel logs, run either of the following commands: $ dmesg # or $ cat /proc/kmsg Copy to clipboard ## Display kernel logs since boot up To display the kernel logs since boot up, run either of the following commands: $ cat /var/log/messages or $ cat /var/log/kern.log Copy to clipboard ## Debug with printk Use `printk` technique to debug in the Linux kernel for printing messages and tracing. The wrappers around printk defined in `include/linux/printk.h`, supports adding a log level to your log statements. For example, pr_emerg("At line %d: Func: %s\n", __LINE__, __func__); pr_info("At line"); Copy to clipboard For more information, see [Message logging with printk](https://www.kernel.org/doc/html/next/core-api/printk-basics.html). ## Use the log levels You can print messages using any other log levels defined in `/linux/printk.h`. The console logs are controlled using the log level used in printk and the log level chosen on the device. Log level should be chosen according to your use case. You may encounter numerous logs if a lower log level is chosen in a frequently called function. The following example shows the logs for kernel print levels: pr_info("At func %s\n", __func__); pr_notice("At func %s\n", __func__); pr_warn("At func %s\n", __func__); pr_err("At func %s\n", __func__); pr_crit("At func %s\n", __func__); pr_alert("At func %s\n", __func__); pr_emerg("At func %s\n", __func__); Copy to clipboard ## Enable SSH Enable a secure shell (SSH) in permissive mode to securely access your host device. For instructions, see [Use SSH](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-254/how_to.html#use-ssh). ## Retrieve information from device You can retrieve the information from your device to debug the kernel issues. The following is the list of kernel commands: Table: List of kernel commands | Commands | Description | | --- | --- | | $ zcat /proc/config.gz
Copy to clipboard | Kernel configuration | | $ cat /proc/cmdline
Copy to clipboard | Kernel command line parameters | | $ cat /proc/version
Copy to clipboard | Kernel version | | $ ls /proc/device-tree
Copy to clipboard | Device tree configuration on device | | $ lsmod
Copy to clipboard | Loaded modules | | $ cat /proc/interrupts
Copy to clipboard | Status of interrupts | | $ cat /sys/devices/system/cpu/cpufreq/policyX/scaling_available_frequecies
Copy to clipboard | Available CPU frequencies where X = cluster | | $ cat /sys/kernel/debug/qcom_socinfo/chip_id
Copy to clipboard | chip\_id using debugfs. | For more information on `procfs`, see [The /proc Filesystem](https://docs.kernel.org/filesystems/proc.html). ## Enable debugfs Debugfs is a file system that allows kernel developers to provide kernel information to the user space. Debugfs is used to access kernel data structures and variables, trace events, debug messages, and statistics. - Ensure that your kernel configuration has `CONFIG_DEBUG_FS` set (enabled by default). - If not, you can enable it in your kernel configuration using `menuconfig`. - Mount debugfs if not already mounted. $ mount -t debugfs none /sys/kernel/debug Copy to clipboard ## Debug kernel and modules through KGDB Kernel GNU debugger (KGDB) provides live debugging support for the kernel on a device. You can configure KGDB to debug core kernel and module issues. KGDB is a source-level debugger used with GDB to debug the Qualcomm Linux kernel. For information on KGDB kernel debugging, see [Debugging kernel and modules via gdb](https://docs.kernel.org/dev-tools/gdb-kernel-debugging.html). ### Use KGDB with serial COM port Enable the following kernel drivers in the kernel configuration file to support KGDB with serial COM port: CONFIG_FRAME_POINTER CONFIG_KGDB CONFIG_KGDB_SERIAL_CONSOLE CONFIG_HAVE_ARCH_KGDB CONFIG_CONSOLE_POLL CONFIG_MAGIC_SYSRQ Copy to clipboard Update the command line for debug: # Append KERNEL_CMDLINE_EXTRA in meta-qcom-hwe/conf/machine/include/qcom-.conf with following string "kgdboc=ttyMSM0,115200n8 kgdbwait nokaslr" # If waiting during the boot of kernel for gdb connection is not desired, # the parameter 'kgdbwait' can be skipped. Copy to clipboard **Disable Watchdog** - Disable the watchdog when debugging using KGDB. The SoC resets when the watchdog is enabled. - Add the following kernel command line parameter to disable the watchdog: echo 1 > /sys/bus/platform/devices/hypervisor:qcom,gh-watchdog/disable Copy to clipboard **Methods to enter debug mode** **Method 1** - Use `kgdbwait`: If `kgdbwait` is passed as the command line parameter, the kernel pauses the execution during boot and enters Debug mode and waits for the connection from remote GDB, with the following message: [ 0.239669] printk: console [ttyMSM0] enabled [ 1.535669] random: fast init done [ 1.541411] KGDB: Registered I/O driver kgdboc [ 2.224804] KGDB: Waiting for connection from remote gdb... Copy to clipboard Disable the watchdog from the configuration. **Method 2** - Use `Magic SysRq`: To enter the Debug mode, run `SysRq-G` in the shell, and run the following command: `echo g > /proc/sysrq-trigger` **Method 3** - Whenever you want to hit a KGDB breakpoint in the source code. Add the following code snippet to the intended driver file. #include (you may need to include this file ) kgdb_breakpoint(); //call this for adding a break point at compile time Copy to clipboard **Method 4** - Triggering panic: The kernel automatically enters Debug mode when an exception occurs. To trigger a crash, use `Magic SysRq` feature. `echo c > /proc/sysrq-trigger` ### Connect to the Qualcomm SoC through GDB over COM Port Install GDB `gdb-multiarch: mingw64\bin\gdb-multiarch.exe`. - GDB for Windows: Install Mingw-w64 through [MSYS2](https://www.msys2.org/). - GDB for Linux: Install GDB using [How to Install GDB](https://www.gdbtutorial.com/tutorial/how-install-gdb). - All other connections to a serial port must be closed, while using COM Port with GDB to connect to the SoC. - On a Windows host - COM Port number must be set to greater than 16. You can get the port number from the device manager. - On a Windows host - COM Port should be passed as `\\.\com<#>`. - In the following example, connect from a serial port on a Windows host. The serial port number is COM17. GDB is disconnected from COM Port before connecting PuTTY again. To attach and de-attach GDB commands, run the following commands: gdb-multiarch.exe -b 115200 (gdb) target remote \\.\com17 Remote debugging using \\.\com17 warning: multi-threaded target stopped without sending a thread-id, using first non-exited thread [Switching to Thread -2] arch_kgdb_breakpoint () at arch/arm64/include/asm/kgdb.h:21 21 arch/arm64/include/asm/kgdb.h: No such file or directory. Copy to clipboard To set the breakpoint, view the breakpoints, or view the CPU registers, use the following commands: # backtrace: > bt # set a break point: break or break : line no # View all current break points: info break # View CPU registers: info reg # step over: s # continue execution: c Copy to clipboard ### Use KGDB to debug kernel modules KGDB is a kernel patch that allows source-level debugging of a running Qualcomm Linux kernel using the GDB interface. You must have the `.ko` files and location of the `.text` kernel module files. To get the address of the `.text` section of the loaded module, run the following command: # from live device # cat /sys/module//sections/.text Copy to clipboard To add a module symbol file, run the following command: (gdb) add-symbol-file <.text_addr> Copy to clipboard For example, `add-symbol-file /sys/modules/linux/linux.ko 0xc0ae22d0`. ## Function trace The ftrace provides tracing utilities that do system-wide profiling and tracing at runtime. For more information on ftrace, see [Function trace](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-12/debugging_linux_kernel.html#sub$Function_trace_ftrace). ### Enable memory-mapped input/output traces The memory mapped I/O (MMIO) traces in the Qualcomm Linux kernel provide information to understand how drivers interact with MMIO devices and their associated hardware states. Generic MMIO traces use `__raw_{read,write}{b,l,w,q}` accessors to read/write from/to memory-mapped registers. In the following cases, the device hangs or some undefined behavior leads to device crashes: > > > Table: Reasons of device crashes > > > | Reasons | Explanation | > | --- | --- | > | Unclocked access | If the access to the register space is unclocked, the device crashes. | > | Protected register space | If the register space is protected and not set for access to the nonsecure world exceptions happen. For example, only exception level (EL3) access is allowed and any EL2/EL1 access is forbidden. | > | xPU control | xPU memory protection units control access to certain memory or register regions for specific clients. | The previous scenarios result in instant reboot, synchronous errors (SErrors)/network on chip (NoC) issues, or interconnect hangs. The `CONFIG_TRACE_MMIO_ACCESS` provides ftrace trace events to log such MMIO register accesses, offering early enablement of trace events, filtering capability, and the ability to dump the ftrace logs on console. The following is the sample output from the trace buffer when the MMIO traces are enabled: # List all rwmmio trace events $ cat /sys/kernel/tracing/available_events | grep rwmmio # Enable all rwmmio trace events $ cat /sys/kernel/tracing/available_events | grep rwmmio >> /sys/kernel/tracing/set_event # List all traces $ cat /sys/kernel/tracing/trace rwmmio_read: gic_peek_irq+0xd0/0xd8 readl addr=0xffff800010040104 rwmmio_write: gic_poke_irq+0xe4/0xf0 writel addr=0xffff800010040184 rwmmio_read: gic_do_wait_for_rwp+0x54/0x90 readl addr=0xffff800010040000 rwmmio_write: gic_set_affinity+0x1bc/0x1e8 writeq addr=0xffff800010046130 Copy to clipboard ### Kprobes Kprobes allows you to break into any kernel routine and collect debugging and performance information nondisruptively. Kernel code address is trapped using a handler routine to invoke when the breakpoint is hit. Kprobes are useful during scheduler debug when you generate traces at different scheduling events to understand the system behavior. For more information, see [Kernel Probes](https://www.kernel.org/doc/html/next/trace/kprobes.html). ## Troubleshoot kernel problems To troubleshoot the kernel issues, use the following methods: ### Boot failure due to incorrect or no DTB picked When troubleshooting boot failure due to DTB load issues, verify authentication and DTB availability, and follow the steps to fix the problem. The following scenarios lead to boot failure: - **Authentication failure** The following example shows the logs for failure due to incorrect authentication: Platform Subtype : 0 DtPlatformLoadDtb qcs6490-rb3gen2.dtb is loaded Platform Subtype : -2090817768 DtPlatformLoadSign qcs6490-rb3gen2.sgn is loaded failed to authenticate image ! Copy to clipboard - **Failure due to unavailability of DTB** The following example shows the logs for failure due to the missing DTB: DtPlatformLoadDtb qcs6490-rb3gen2.dtb is loading failed with Status = E DtPlatformDxeEntryPoint: no DTB blob could be loaded, defaulting to ACPI (Status == Not Found) Copy to clipboard To fix the unavailability of DTB issues, use the following steps: - Verify if the corresponding DTB file is a part of a packaged `efi.bin` file. - Mount the `efi.bin` file locally on the host development machine to verify the following: $ mount -t vfat efi.bin /mnt/ $ ls -lR /mnt/ Copy to clipboard ### Serial console not working To troubleshoot serial console log failure, enable specific drivers in the kernel configuration and add relevant parameters to the kernel boot arguments. - Enable the following driver in the kernel configuration file: - `CONFIG_SERIAL_QCOM_GENI=y` - `CONFIG_SERIAL_QCOM_GENI_CONSOLE=y` - Add the following parameter to the kernel boot arguments: console=ttyMSM0,115200n8 Copy to clipboard - To get early boot messages from the kernel, add the following parameter to the kernel boot arguments: earlycon Copy to clipboard ### Debug Remoteproc failure To troubleshoot remoteproc failure, capture kernel logs, use matching firmware signature files and verify firmware locations in the device. - Ensure that all the matching subsystem images are properly flashed without any errors. - Verify if there are any errors in the kernel log. The following example shows a sample kernel log: 0x000000000A27652C | 5198.790423: qcom_q6v5_pas 3000000.remoteproc: fatal error received: err_inject_crash.c:413:Crash injected via Diag 0x000000000A276689 | 5198.801061: remoteproc remoteproc2: crash detected in 3000000.remoteproc: type fatal error 0x000000000A2767A1 | 5198.809602: remoteproc remoteproc2: handling crash #1 in 3000000.remoteproc 0x000000000A27688E | 5198.816837: remoteproc remoteproc2: recovering 3000000.remoteproc 0x000000000A276971 | 5198.823784: qcom_q6v5_pas 8a00000.remoteproc: subsystem event rejected Copy to clipboard - Disable the following subsystem restart feature to ensure that the remoteproc crash signature is visible in the kernel log: echo disabled > /sys/kernel/debug/remoteproc/remoteprocN/recovery Copy to clipboard - Confirm if all the necessary firmware files are present in the `/lib/firmware/qcom/ in rootfs` file system. ### Debug configurations or symbols not reflecting in images To debug kernel configuration change issues where configurations or symbols are not reflecting in the images, follow these steps: - Add the debug configuration driver to the `arch/arm64/configs/qcom_debug.config` file. - Export the macro `DEBUG_BUILD=1` before running the `bitbake` command. Note The previous debug configuration file pertains to a custom BSP variant. Add the changes to `qcom.cfg` file for the base BSP variant. ### System memory is too low To debug the cases where a system runs out of free memory, use the procedure mentioned in the [Out of memory](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-12/debugging_linux_kernel.html#sub$Out_of_memory). ### Identify DTB during boot When the device boots, the bootloader verifies the following IDs and loads the corresponding DTB file: qcom,msm-id = ; qcom,board-id = ; Copy to clipboard - x = ID for SoC - z = ID for SoC revision (reserved field) - y = ID for CDP, MTP (hardware variants), and Platform - y’ = ID for Subtype (assumed 0 if absent) Last Published: Oct 15, 2024 [Previous Topic Customize](https://docs.qualcomm.com/bundle/publicresource/80-70015-3/topics/customize.md) [Next Topic References](https://docs.qualcomm.com/bundle/publicresource/80-70015-3/topics/references.md)