# Debug Source: [https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html](https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html) Use the following methods to understand the high-level categorization of ways to debug the kernel, and report issues identified with the inherent debug features in the Linux kernel. For more information on debug, see [Qualcomm Linux Debug Guide](bundle/publicresource/topics/80-70014-12). ### Serial console Standard boot and kernel logs are accessible using a serial console. The serial console enables live debugging on a live target. Qualcomm kernel enables the `CONFIG_SERIAL_QCOM_GENI` driver to support UART and console. Add the following line to the kernel command line parameter: console=ttyMSM0,115200n8 # as in following line in meta-qcom-hwe/conf/machine/qcm6490.conf KERNEL_CMDLINE_EXTRA ?= "root=/dev/disk/by-partlabel/system rw rootwait console=ttyMSM0,115200n8 pcie_pme=nomsi earlycon" Copy to clipboard When you rebuild and load the kernel: - Connect a Micro USB to USB-A type cable as follows: - Device > Micro USB====USB-TypeA > Host machine - To connect to a device serial port on the host, use a serial console client on a host machine. - You should be able to see serial console logs and other kernel logs on serial client. For more information on the serial console, see [Linux Serial Console](https://docs.kernel.org/admin-guide/serial-console.html). ### Configure console log level Configure the kernel console log level for extensive or minimal logging to satisfy debug or release software versions. Use `/proc/sys/kernel/printk` to configure log level to control log messages appearing on the console. Set the log level from 1 to 7. When you set the log level value to 1, it filters the log to the lowest level. When the log level value is set to 1, only` pr_emerg/KERN_EMERG` printk logs are printed. When you set the log level value to 7, it enables the highest log level `pr_info/KERN_INFO` and prints all printk logs to the console. $ echo "1" > /proc/sys/kernel/printkCopy to clipboard ### Display kernel logs Run either of the following commands to display the kernel logs: $ dmesg # or $ cat /proc/kmsgCopy to clipboard ### Display kernel logs since boot up Run either of the following commands to display the kernel logs since boot up: $ cat /var/log/messages or $ cat /var/log/kern.logCopy to clipboard ### Debugging with printk Debugging with `printk` is a technique in the Linux kernel for printing messages and tracing. The wrappers around printk defined in `include/linux/printk.h`, supports adding a log level to your log statements. For example, pr_emerg("At line %d: Func: %s\n", __LINE__, __func__); pr_info("At line");Copy to clipboard For more information, see [Message logging with printk](https://www.kernel.org/doc/html/next/core-api/printk-basics.html). ### Using the log levels You can print messages using any other log levels defined in `/linux/printk.h`. The console logs are controlled using the log level used in printk and the log level chosen on the device. Log level should be chosen with each case according to your use case. You can run into a lot of logs if a lower log level is chosen in a frequently called function. The following example shows the logs for kernel print levels: pr_info("At func %s\n", __func__); pr_notice("At func %s\n", __func__); pr_warn("At func %s\n", __func__); pr_err("At func %s\n", __func__); pr_crit("At func %s\n", __func__); pr_alert("At func %s\n", __func__); pr_emerg("At func %s\n", __func__);Copy to clipboard ## Enable SSH Source: [https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html](https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html) Enable secure shell (SSH) in permissive mode to securely access your host device. For instructions, see [How to SSH?](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-254/how_to.html#how-to-ssh-). ## Retrieve information from device Source: [https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html](https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html) You can retrieve the information from your device to debug the kernel issues. The following is the list of kernel commands: Table : List of kernel commands | Commands | Description | | --- | --- | | $ zcat /proc/config.gzCopy to clipboard | Kernel configuration | | $ cat /proc/cmdlineCopy to clipboard | Kernel command line parameters | | $ cat /proc/versionCopy to clipboard | Kernel version | | $ ls /proc/device-treeCopy to clipboard | Device-tree configuration on device | | $ lsmodCopy to clipboard | Loaded modules | | $ cat /proc/interruptsCopy to clipboard | Status of interrupts | | $ cat /sys/devices/system/cpu/cpufreq/policyX/scaling_available_frequeciesCopy to clipboard | Available CPU frequencies where X = cluster | | $ cat /sys/kernel/debug/qcom_socinfo/chip_idCopy to clipboard | chip\_id using [debugfs](https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html#enable-debugfs). | For more information on `procfs`, see [The /proc Filesystem](https://docs.kernel.org/filesystems/proc.html). ## Enable debugfs Source: [https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html](https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html) Debugfs is a file system that allows kernel developers to provide kernel information to the user space. Debugfs is used to access kernel data structures and variables, trace events, debug messages, and statistics. - Ensure that your kernel configuration has `CONFIG_DEBUG_FS` set (enabled by default). - If not, you can enable it in your kernel configuration using `menuconfig`. - Mount debugfs if not already mounted. $ mount -t debugfs none /sys/kernel/debugCopy to clipboard ## Debugging kernel and modules through KGDB Source: [https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html](https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html) Kernel GNU debugger (KGDB) provides live debugging support for the kernel on a target. You can configure KGDB to debug core kernel and module issues. KGDB is a source-level debugger used with GDB to debug the Linux kernel. For information on KGDB kernel debugging, see [Debugging kernel and modules via gdb](https://docs.kernel.org/dev-tools/gdb-kernel-debugging.html). ### Use KGDB with serial COM port Enable the following kernel drivers in the kernel configuration file to support KGDB with serial COM port: CONFIG_FRAME_POINTER CONFIG_KGDB CONFIG_KGDB_SERIAL_CONSOLE CONFIG_HAVE_ARCH_KGDB CONFIG_CONSOLE_POLL CONFIG_MAGIC_SYSRQCopy to clipboard Update the command line for debug: # Append KERNEL_CMDLINE_EXTRA in meta-qcom-hwe/conf/machine/qcm6490.conf with following string "kgdboc=ttyMSM0,115200n8 kgdbwait nokaslr" # If waiting during the boot of kernel for gdb connection is not desired, # the parameter 'kgdbwait' can be skipped.Copy to clipboard **Disable Watchdog** - Disable the watchdog when debugging using KGDB. The SoC resets when the watchdog is enabled. - Add the following kernel command line parameter to disable the watchdog: echo 1 > /sys/bus/platform/devices/hypervisor:qcom,gh-watchdog/disableCopy to clipboard **Ways to enter debug mode** **Method 1** - Use `kgdbwait`: If `kgdbwait` is passed as the command line parameter, the kernel pauses the execution during boot and enters Debug mode and waits for the connection from remote GDB, with the following message: [ 0.239669] printk: console [ttyMSM0] enabled [ 1.535669] random: fast init done [ 1.541411] KGDB: Registered I/O driver kgdboc [ 2.224804] KGDB: Waiting for connection from remote gdb...Copy to clipboard Disable the watchdog from the configuration. **Method 2** - Use `Magic SysRq`: To enter the Debug mode, run `SysRq-G` in the shell, and execute the following command: `echo g > /proc/sysrq-trigger` **Method 3** - Whenever you want to hit a KGDB breakpoint in the source code. Add the following code snippet to the intended driver file. #include (you may need to include this file ) kgdb_breakpoint(); //call this for adding a break point at compile timeCopy to clipboard **Method 4** - Triggering panic: The kernel automatically enters Debug mode when an exception occurs. To trigger a crash, use `Magic SysRq` feature. `echo c > /proc/sysrq-trigger` ### Connecting to the target (SoC) through GDB over COM Port Install GDB `gdb-multiarch: mingw64\bin\gdb-multiarch.exe`. - GDB for Windows: Install Mingw-w64 through [MSYS2](https://www.msys2.org/). - GDB for Linux: Install GDB using [How to Install GDB](https://www.gdbtutorial.com/tutorial/how-install-gdb). - All other connections to a serial port must be closed, while using COM Port with GDB to connect to the SoC. - On a Windows host - COM Port number must be set to greater than 16. You can get the port number from the device manager. - On a Windows host - COM Port should be passed as `\\.\com<#>` - In the following example, connect from a serial port on a Windows host. The serial port number is COM17. GDB is disconnected from COM Port before connecting PuTTY again. To attach and de-attach GDB commands, run the following commands: gdb-multiarch.exe -b 115200 (gdb) target remote \\.\com17 Remote debugging using \\.\com17 warning: multi-threaded target stopped without sending a thread-id, using first non-exited thread [Switching to Thread -2] arch_kgdb_breakpoint () at arch/arm64/include/asm/kgdb.h:21 21 arch/arm64/include/asm/kgdb.h: No such file or directory.Copy to clipboard Use the following commands to set the breakpoint to view the breakpoints, or to view the CPU registers: # backtrace: > bt # set a break point: break or break : line no # View all current break points: info break # View CPU registers: info reg # step over: s # continue execution: cCopy to clipboard ### Using KGDB to debug kernel modules KGDB is a kernel patch that enables source-level debugging of a running Linux kernel using the GDB interface. You must have the `.ko` files and location of the `.text` kernel module files. To get the address of the `.text` section of the loaded module, run: # from live target # cat /sys/module//sections/.textCopy to clipboard To add a module symbol file, run: (gdb) add-symbol-file <.text_addr>Copy to clipboard For example, `add-symbol-file /sys/modules/linux/linux.ko 0xc0ae22d0`. ## Function trace Source: [https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html](https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html) The ftrace provides tracing utilities that do system-wide profiling and tracing at runtime. For more information on Ftrace, see [Function trace](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-12/debugging_linux_kernel.html#sub$Function_trace_ftrace). ### Enable memory-mapped input/output traces Source: [https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html](https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html) The memory mapped I/O (MMIO) traces in the Linux kernel provide information to understand how drivers interact with MMIO devices and their associated hardware states. Generic MMIO traces use `__raw_{read,write}{b,l,w,q}` accessors to read/write from/to memory-mapped registers. In the following cases, the device hangs or some undefined behavior leads to device crashes: Table : Reasons of device crashes | Reasons | Explanation | | --- | --- | | Unclocked access | If the access to the register space is unclocked, the device
crashes. | | Protected register space | If the register space is protected, and not set for access to the
nonsecure world exceptions happen.
For example, only exception level
(EL3) access is allowed and any EL2/EL1 access is
forbidden. | | xPU control | xPU memory protection units control access to certain memory or
register regions for specific clients. | The previous scenarios result in instant reboot/synchronous errors (SErrors)/network on chip (NoC), or interconnect hang. The `CONFIG_TRACE_MMIO_ACCESS` provides ftrace trace events to log such MMIO register accesses, which offer early enablement of trace events, filtering capability, and dumping the ftrace logs on console. The following is the sample output from the trace buffer when the MMIO traces are enabled: # List all rwmmio trace events $ cat /sys/kernel/tracing/available_events | grep rwmmio # Enable all rwmmio trace events $ cat /sys/kernel/tracing/available_events | grep rwmmio >> /sys/kernel/tracing/set_event # List all traces $ cat /sys/kernel/tracing/trace rwmmio_read: gic_peek_irq+0xd0/0xd8 readl addr=0xffff800010040104 rwmmio_write: gic_poke_irq+0xe4/0xf0 writel addr=0xffff800010040184 rwmmio_read: gic_do_wait_for_rwp+0x54/0x90 readl addr=0xffff800010040000 rwmmio_write: gic_set_affinity+0x1bc/0x1e8 writeq addr=0xffff800010046130Copy to clipboard ### Kprobes Source: [https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html](https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html) Kprobes enables you to break into any kernel routine and collect debugging and performance information nondisruptively. Kernel code address is trapped using a handler routine to invoke when the breakpoint is hit. Kprobes are useful during scheduler debug when you generate traces at different scheduling events to understand the system behavior. For more information, see [Kernel Probes](https://www.kernel.org/doc/html/next/trace/kprobes.html). ## Troubleshoot kernel problems Source: [https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html](https://docs.qualcomm.com/doc/80-70014-3/topic/debug.html) Use the following methods to troubleshoot the kernel issues: ### Boot failure due to incorrect or no DTB picked When troubleshooting boot failure due to DTB load issues, verify authentication and DTB availability, and follow the steps to fix the problem. The following scenarios lead to boot failure: - **Authentication failure** The following example shows the logs for failure due to incorrect authentication: Platform Subtype : 0 DtPlatformLoadDtb qcm6490-rb3gen2.dtb is loaded Platform Subtype : -2090817768 DtPlatformLoadSign qcm6490-rb3gen2.sgn is loaded failed to authenticate image !Copy to clipboard - **Failure due to unavailability of DTB** The following example shows the logs for failure due to the missing DTB: DtPlatformLoadDtb qcm6490-rb3gen2.dtb is loading failed with Status = E DtPlatformDxeEntryPoint: no DTB blob could be loaded, defaulting to ACPI (Status == Not Found)Copy to clipboard Use the following steps to fix the unavailability of DTB issues: - Verify if the corresponding DTB file is a part of a packaged `efi.bin` file. - Mount `efi.bin` file locally on the host development machine to verify the following: $ mount -t vfat efi.bin /mnt/ $ ls -lR /mnt/Copy to clipboard ### Serial console not working To troubleshoot serial console log failure, enable specific drivers in the kernel configuration and add relevant parameters to the kernel boot arguments. - Enable the following driver in the kernel configuration file: - `CONFIG_SERIAL_QCOM_GENI=y` - `CONFIG_SERIAL_QCOM_GENI_CONSOLE=y` - Add the following parameter to the kernel boot arguments: console=ttyMSM0,115200n8 Copy to clipboard - To get early boot messages from the kernel, add the following parameter to the kernel boot arguments: earlyconCopy to clipboard ### Remoteproc not coming up To troubleshoot remoteproc failure, capture kernel logs, use matching firmware signature files and check firmware locations in the device. - Ensure that all the matching subsystem images are properly flashed without any errors. - Verify if there are any errors in the kernel log. The following example shows a sample kernel log: 0x000000000A27652C | 5198.790423: qcom_q6v5_pas 3000000.remoteproc: fatal error received: err_inject_crash.c:413:Crash injected via Diag 0x000000000A276689 | 5198.801061: remoteproc remoteproc2: crash detected in 3000000.remoteproc: type fatal error 0x000000000A2767A1 | 5198.809602: remoteproc remoteproc2: handling crash #1 in 3000000.remoteproc 0x000000000A27688E | 5198.816837: remoteproc remoteproc2: recovering 3000000.remoteproc 0x000000000A276971 | 5198.823784: qcom_q6v5_pas 8a00000.remoteproc: subsystem event rejectedCopy to clipboard - Disable the following subsystem restart feature to ensure that the remoteproc crash signature is visible in the kernel log: echo disabled > /sys/kernel/debug/remoteproc/remoteprocN/recovery Copy to clipboard - Confirm if all the necessary firmware files are present in the `/lib/firmware/qcom/qcm6490 in rootfs` file system. ### Debug configurations or symbols not reflecting in images To debug kernel configuration change issues where configurations or symbols are not reflecting in the images, follow these steps: - Add the debug configuration driver to the `arch/arm64/configs/qcom_debug.config` file. - Export the macro `DEBUG_BUILD=1` before executing the `bitbake` command. ### System memory is too low Use the procedure mentioned in the following link to debug cases where a system runs out of free memory. For more information on memory, see [Out of memory](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-12/debugging_linux_kernel.html#sub$Out_of_memory). Last Published: Jul 15, 2024 [Previous Topic Customize](https://docs.qualcomm.com/bundle/publicresource/80-70014-3/topics/customize.md) [Next Topic References](https://docs.qualcomm.com/bundle/publicresource/80-70014-3/topics/references.md)