# Debug Linux kernel space
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
The following logs, command line options, and dumps can be used to debug issues in kernel
space.
Page-1 Dynamic debug Dynamic debug /proc file system /proc file system RAMdump RAMdump RAMparser RAMparser Logs Logs Command line Command line Kernel debugging configuration Kernel debugging configuration Dumps Dumps Function trace Function trace /sys/kernel/debug debugfs file system /sys/kernel/debug debugfs file system Crash utility Crash utility
At a high-level, issues in the kernel space are categorized as panic and bugs. You can
generate kernel logs using the `dmesg` command.
It is recommended to use debug build for debugging kernel issues. For more information on
how to generate the debug build, see [Overview of metadata layers](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-27/platform_software_features.html#overview_of_meta_data_layers).
For information on kernel source configuration files, see [Qualcomm Linux Kernel Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-3/overview.html).
## Enable dynamic debug
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
### About this task
The debugfs file system allows you to debug the kernel by enabling logs at runtime.
Enabling logs statically makes the system slow. When verifying a particular scenario,
you can use the debugfs to enable logs for the specific time period.
By default, the dynamic debugfs is disabled. To enable debugfs in the kernel, do the
following:
### Procedure
1. Enable the `CONFIG_DYNAMIC_DEBUG` kernel configuration.
2. Recompile and reflash the build.
3. Mount the debugfs file system.
mount -o rw,remount -t debugfs none /sys/kernel/debugCopy to clipboard
To verify if the dynamic debug is enabled or not, verify if the following node
exists:
cat /sys/kernel/debug/dynamic_debugCopy to clipboard
If the node exists, verify which logs are defined.
cat /sys/kernel/debug/dynamic_debug/controlCopy to clipboard
4. Enable the debug log for the files or the function that requires debugging.
For example,
- The following command enables all debugfs logs in the
mdp.c file:
echo 'file mdp.c +p' > /sys/kernel/debug/dynamic_debug/controlCopy to clipboard
- The following command enables the log at line 2921 in the
mdp.c file:
echo 'file mdp.c line 2921 +p' > /sys/kernel/debug/dynamic_debug/controlCopy to clipboard
5. Verify the logs using `dmesg` or run the command:
cat /proc/kmsgCopy to clipboard
For more information, see [Documentation/dynamic-debug-howto.txt](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/dynamic-debug-howto.rst).
## Enable kernel debugging configuration
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
Kernel configuration options can be enabled to debug different issues. To learn more about
the kernel debug configuration options, run the following
command:
menuconfigCopy to clipboard
Sample output:
Kernel hacking
[*] Kernel debugging
[*] Detect Soft Lockups
[ ] Collect scheduler statistics
[*] Debug slab memory allocations
[*] Memory leak debugging
[*] Mutex debugging, deadlock detection
[*] Spinlock debugging
[*] Sleep-inside-spinlock checking
[ ] kobject debugging
[ ] Highmem debugging
[ ] Compile the kernel with debug info
Copy to clipboard
Note: If you enable any of these configuration options,
the kernel slows down a bit. Hence, if you notice any decrease in performance, you can
disable the options.
## Kernel config options
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
The following table lists various common kernel config options that are useful for
debugging:
| Kernel debug config option | Description |
| --- | --- |
| CONFIG\_DEBUG\_LIST | Turns on checks for performing standard linked list manipulations with list.h header file. If the pointers do not match, a warning is printed followed by `BUG_ON` crash. |
| CONFIG\_PAGE\_POISONING CONFIG\_DEBUG\_PAGEALLOC | Fills the pages with poison patterns `(PAGE_POISON 0xaa)` after calling `free_pages()` and verifies the patterns before calling `alloc_pages()`. |
| CONFIG\_DEBUG\_USER | Prints a message when a user space process is killed due to a segfault or invalid instruction such as `user_debug=31` in the arch/arm/Kconfig.debug file. Kernel boot parameter must be added to the BoardConfig.mk file. |
| CONFIG\_DEBUG\_SPINLOCK | Catches missing spinlock initialization and certain other kinds of common spinlock errors, such as:
Waiting for more than 1 second on a spinlock Freeing an already freed lock Reinitializing a lock that was already used |
| CONFIG\_DEBUG\_MUTEXES | Allows mutex semantics violations to be detected. |
| DEBUG\_LOCK\_ALLOC | Detects incorrect freeing of live locks. |
| CONFIG\_SLUB CONFIG\_SLUB\_DEBUG | Puts additional checks in place to detect corruption of internal kernel memory allocation structures by adding poison for `use-after-free (0x6b)` and `buffer-overflow-padding (0xbb)`. |
| Kernel debug config options for additional debugging that can be verbose and can make the system slow. | Kernel debug config options for additional debugging that can be verbose and can make the system slow. |
| CONFIG\_DEBUG\_ATOMIC\_SLEEP | Various routines that may sleep become noisy when they are called inside atomic sections. |
| DEBUG\_SPINLOCK\_SLEEP | Routines that may sleep become noisy when they are called with a spinlock held. |
| CONFIG\_DEBUG\_VM, CONFIG\_DEBUG\_HIGHMEM | Provides additional debugging support for virtual memory management corruptions. |
| CONFIG\_DEBUG\_OBJECTS | Tracks the lifetime of various objects and validates the operations on those objects. |
Following are example logs indicating some common memory issues.
### List corruptions
When `CONFIG_DEBUG_LIST` option is enabled, the following crash
signatures can be identified:
**Sample crash signature 1**
<4> WARNING: at kernel/lib/list_debug.c:60 __list_del_entry+0xa0/0xd0()
<6> list_del corruption. prev->next should be c6fc374c, but was c18d804c
<6> Modules linked in: adsprpc
<6> [] (unwind_backtrace+0x0/0x138) from [] (warn_slowpath_common+0x4c/0x64)
<6> [] (warn_slowpath_common+0x4c/0x64) from [] (warn_slowpath_fmt+0x30/0x40)
<6> [] (warn_slowpath_fmt+0x30/0x40) from [] (__list_del_entry+0xa0/0xd0)
<6> [] (__list_del_entry+0xa0/0xd0) from [] (account_entity_dequeue+0x84/0x94)
<6> [] (account_entity_dequeue+0x84/0x94) from [] (dequeue_task_fair+0x64/0x190)Copy to clipboard
**Sample crash signature 2**
<4>WARNING: at kernel/lib/list_debug.c:52 __list_del_entry+0x8c/0xac()
<4> list_del corruption, e240c500->prev is LIST_POISON2 (00200200)
<4> Modules linked in: wlan(PO) cfg80211 adsp_loader
<4> [] (unwind_backtrace+0x0/0x11c)
from [] (warn_slowpath_common+0x4c/0x64)
<4> [] (warn_slowpath_common+0x4c/0x64)
from [] (warn_slowpath_fmt+0x2c/0x3c)
<4> [] (warn_slowpath_fmt+0x2c/0x3c)
from [] (__list_del_entry+0x8c/0xac)
<4> [] (__list_del_entry+0x8c/0xac)
from [] (list_del+0xc/0x24)
<4> [] (list_del+0xc/0x24)
from [] (binder_thread_read+0x488/0xb70)
<4> [] (binder_thread_read+0x488/0xb70)
from [] (binder_ioctl+0x220/0x5b8)Copy to clipboard
**Sample crash signature 3**
<4> WARNING: at kernel/lib/list_debug.c:47 __list_del_entry+0x90/0xb0()
<4> list_del corruption, d6ed3720->next is LIST_POISON1 (00100100)
<4> Modules linked in:
<4> [] (unwind_backtrace+0x0/0x12c) from [] (warn_slowpath_common+0x4c/0x64)
<4> [] (warn_slowpath_common+0x4c/0x64) from [] (warn_slowpath_fmt+0x2c/0x3c)
<4> [] (warn_slowpath_fmt+0x2c/0x3c) from [] (__list_del_entry+0x90/0xb0)
<4> [] (__list_del_entry+0x90/0xb0) from [] (list_del+0xc/0x28)
<4> [] (list_del+0xc/0x28) from [] (bam_mux_write_done+0x34/0x11c)Copy to clipboard
### Spinlock corruption issues
When `CONFIG_DEBUG_SPINLOCK` and `CONFIG_DEBUG_MUTEXES`
kernel configuration options are enabled, the following crash signatures can be
identified.
**Crash signature 1**
<0>BUG: spinlock lockup on CPU#1, ndroid.launcher/1071
<0> lock: 0xd5e8f480, .magic: dead4ead, .owner: /-1, .owner_cpu: -1
<4> [] (spin_dump+0x74/0x84) from []
(do_raw_spin_lock+0x144/0x188)
<4> [] (do_raw_spin_lock+0x144/0x188) from [] (kgsl_mmu_pt_get_flags+0x18/0x44)
<4> [] (kgsl_mmu_pt_get_flags+0x18/0x44) from [] (adreno_ringbuffer_submitcmd+0x168/0x228)
<4> [] (adreno_ringbuffer_submitcmd+0x168/0x228) from [] (sendcmd+0x3c/0x254)Copy to clipboard
**Crash signature 2**
<0> BUG: spinlock lockup on CPU#2, kworker/2:0H/2910
<0> lock: kpss_clock_reg_lock+0x0/0x10, .magic: dead4ead, .owner:
kworker/3:0H/2904, .owner_cpu: 3
<6> kworker/2:0H (2910): undefined instruction: pc=c0963098
kernel BUG at kernel/lib/spinlock_debug.c:95!
<4> [] (spin_dump+0x7c/0x94) from []
(do_raw_spin_lock+0xcc/0x164)
<4> [] (do_raw_spin_lock+0xcc/0x164) from [] (_raw_spin_lock_irqsave+0x20/0x28)
<4> [] (_raw_spin_lock_irqsave+0x20/0x28) from [] (__kpss_mux_set_sel+0x14/0x80)
<4> [] (__kpss_mux_set_sel+0x14/0x80) from [] (kpss_mux_set_sel+0x18/0x20)Copy to clipboard
### Slub poisoning issues
When `CONFIG_SLUB` and `CONFIG_SLUB_DEBUG` kernel
configuration options are enabled, the following crash signature can be
identified:
<3>[ 3438.930472] =============================================================================
<3>[ 3438.937628] BUG kmalloc-64 (Tainted: G W O): Poison overwritten
<3>[ 3438.944223] -----------------------------------------------------------------------------
<3>[ 3438.944228]
<3>[ 3438.953861] INFO: 0xce308408-0xce30840b. First byte 0x0 instead of 0x6b
<3>[ 3438.960470] INFO: Allocated in kgsl_ioctl_drawctxt_create+0x2c/0x2ac age=4426 cpu=0 pid=21702
<3>[ 3438.968970] __slab_alloc.isra.37.constprop.43+0x4d4/0x534
<3>[ 3438.974435] kmem_cache_alloc_trace+0x240/0x258
<3>[ 3438.978947] kgsl_ioctl_drawctxt_create+0x2c/0x2ac
<3>[ 3438.983721] kgsl_ioctl+0xfc/0x324
<3>[ 3438.987108] do_vfs_ioctl+0x80/0x54c
<3>[ 3438.990669] sys_ioctl+0x38/0x5c
<3>[ 3438.993880] ret_fast_syscall+0x0/0x30
<3>[ 3438.997615] INFO: Freed in kgsl_release+0xb8/0xc0 age=10 cpu=3 pid=21842
<3>[ 3439.004298] __slab_free+0x30/0x308
<3>[ 3439.007770] kgsl_release+0xb8/0xc0
<3>[ 3439.011242] fput+0xcc/0x23c
<3>[ 3439.014106] filp_close+0x68/0x80
<3>[ 3439.017407] put_files_struct+0xd8/0x110
<3>[ 3439.021312] do_exit+0x164/0x860
<3>[ 3439.024524] do_group_exit+0x3c/0xb0
<3>[ 3439.028084] get_signal_to_deliver+0x2c4/0x59c
<3>[ 3439.032510] do_signal+0x90/0x480
<3>[ 3439.035809] do_notify_resume+0x50/0x5c
<3>[ 3439.039629] work_pending+0x24/0x28
<3>[ 3439.043101] INFO: Slab 0xc0fbcd20 objects=16 used=16 fp=0x (null) flags=0x0080
<3>[ 3439.050401] INFO: Object 0xce308400 @offset=1024 fp=0xce308900
<3>[ 3439.050405]
<3>[ 3439.057694] Bytes b4 ce3083f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
<3>[ 3439.066457] Object ce308400: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b kkkkkkkk....kkkk
<3>[ 3439.075051] Object ce308410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
<3>[ 3439.083639] Object ce308420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
<3>[ 3439.092234] Object ce308430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
<3>[ 3439.100828] Redzone ce308440: bb bb bb bb ....
<3>[ 3439.108463] Padding ce3084e8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
<3>[ 3439.117145] Padding ce3084f8: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
<4>[ 3439.125144] [] (unwind_backtrace+0x0/0x138) from [] (check_bytes_and_report+0xc0/0xe4)
<4>[ 3439.134771] [] (check_bytes_and_report+0xc0/0xe4) from [] (check_object+0x1b8/0x214)
<4>[ 3439.144237] [] (check_object+0x1b8/0x214) from [] (alloc_debug_processing+0x7c/0x150)
<4>[ 3439.153782] [] (alloc_debug_processing+0x7c/0x150) from [] (__slab_alloc.isra.37.constprop.43+0x4d4/0x534)
<4>[ 3439.165153] [] (__slab_alloc.isra.37.constprop.43+0x4d4/0x534) from [] (kmem_cache_alloc_trace+0x240/0x258)
<4>[ 3439.176611] [] (kmem_cache_alloc_trace+0x240/0x258) from [] (__get_vm_area_node.isra.26+0x84/0x174)
<4>[ 3439.187376] [] (__get_vm_area_node.isra.26+0x84/0x174) from [] (get_vm_area_caller+0x44/0x4c)
<4>[ 3439.197615] [] (get_vm_area_caller+0x44/0x4c) from [] (vmap+0x50/0x90)
<4>[ 3439.205866] [] (vmap+0x50/0x90) from [] (_kgsl_sharedmem_page_alloc+0x238/0x3d4)
<4>[ 3439.214979] [] (_kgsl_sharedmem_page_alloc+0x238/0x3d4) from [] (_gpumem_alloc+0xb0/0xfc)
<4>[ 3439.224874] [] (_gpumem_alloc+0xb0/0xfc) from [] (kgsl_ioctl_gpumem_alloc_id+0x40/0x1a8)
<4>[ 3439.234684] [] (kgsl_ioctl_gpumem_alloc_id+0x40/0x1a8) from [] (kgsl_ioctl+0xfc/0x324)
<4>[ 3439.244319] [] (kgsl_ioctl+0xfc/0x324) from [] (do_vfs_ioctl+0x80/0x54c)
<4>[ 3439.252738] [] (do_vfs_ioctl+0x80/0x54c) from [] (sys_ioctl+0x38/0x5c)
<4>[ 3439.260986] [] (sys_ioctl+0x38/0x5c) from [] (ret_fast_syscall+0x0/0x30)
<3>[ 3439.269400] FIX kmalloc-64: Restoring 0xce308408-0xce30840b=0x6bCopy to clipboard
### Page poisoning issues
When `CONFIG_DEBUG_PAGEALLOC` and `CONFIG_PAGE_POISONING`
kernel configuration options are enabled, the following crash signature can be
identified:
<1> Unable to handle kernel paging request at virtual address aaaaaaae
<1> pgd = e98b4000
<1> [aaaaaaae] *pgd=00000000
<0> Internal error: Oops: 5 [#1] PREEMPT SMP ARM
<4> Modules linked in: adsp_loader exfat_fs(P) exfat_core(P)
<4> CPU: 1 Tainted: P W (3.4.0-628250-eng #1)
<4> PC is at pid_nr_ns+0xc/0x3c
<4> LR is at do_task_stat+0x248/0x83c
<4> pc : [] lr : [] psr: a0000093Copy to clipboard
## CPU parameters
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
For details about CPU parameters, see [https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-devices-system-cpu](https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-devices-system-cpu).
## Memory
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
Linux is a virtual memory system. Hence, the addresses viewed by the user programs do not
correspond to the physical addresses used by the hardware directly. Virtual memory
introduces a layer of indirection that allows programs running on the system to allocate
far more memory than the physically available memory.
Memory management implementation covers the following areas:
- Management of physical pages in the memory
- Buddy system to allocate memory in large chunks
- Slab, slub, and slob allocators to allocate smaller chunks of memory
- vmalloc mechanism to allocate noncontiguous blocks of memory
- Address space of processes
### /proc file system
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
The following files are available on the /proc file system:
- [/proc/meminfo](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html#zbq1520725114522__section_N10011_N1000E_N10001)
- [/proc/vmstat](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html#zbq1520725114522__section_N1002B_N1000E_N10001)
- [/proc/iomem](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html#zbq1520725114522__section_N10045_N1000E_N10001)
- [/proc/vmallocinfo](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html#zbq1520725114522__section_N10053_N1000E_N10001)
### /proc/meminfo
This file provides information about the distribution and usage of memory. To view the
contents of this file, run the command:
cat proc/meminfoCopy to clipboard
Sample
output:
MemTotal: 3813532 kB
MemFree: 624836 kB
MemAvailable: 2098008 kB
Buffers: 40416 kB
Cached: 1484320 kB
SwapCached: 0 kB
Active: 1334816 kB
.
.
.
.
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2431048 kB
Committed_AS: 99995284 kB
VmallocTotal: 258867136 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
CmaTotal: 163840 kB
CmaFree: 1368 kB
Copy to clipboard
For details about the parameters available in this file, see [https://docs.kernel.org/filesystems/proc.html](https://docs.kernel.org/filesystems/proc.html).
### /proc/vmstat
This file shows detailed virtual memory statistics from the kernel. Most counters are
available only if the `CONFIG_VM_EVENT_COUNTERS` option in the
init/Kconfig file is enabled.
To view the contents of this file, run the command:
cat proc/vmstatCopy to clipboard
Sample output:
nr_free_pages 156290
nr_alloc_batch 132
nr_inactive_anon 108
nr_active_anon 165006
nr_inactive_file 212275
nr_active_file 168709
nr_unevictable 64
nr_mlock 64
nr_anon_pages 164982
nr_mapped 90366
nr_file_pages 381184:
:
unevictable_pgs_mlocked 0
unevictable_pgs_munlocked 0
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 0
Copy to clipboard
For details about the parameters available in this file, see [https://man7.org/linux/man-pages/man8/vmstat.8.html](https://man7.org/linux/man-pages/man8/vmstat.8.html).
### /proc/iomem
This file shows the memory map of the system for its various device drivers. To view the
contents of this file, run the command:
cat proc/iomemCopy to clipboard
Sample output:
007781b8-007791b7 : vmpm
010aa000-010abfff : tsens_physical
010ac000-010ac003 : pshold-base
010ad000-010aefff : tsens_physical
01680000-0168ffff : /soc/arm,smmu-anoc1@1680000
016c0000-016fffff : /soc/arm,smmu-anoc2@16c0000
01d0101c-01d0101f : sp2soc_irq_status
01d01024-01d01027 : sp2soc_irq_clr
01d01028-01d0102b : sp2soc_irq_mask
.
.
.
0caa0000-0caa3fff : jpeg_hw
0caa4000-0caa47ff : fd_core
0caa5000-0caa53ff : fd_misc
0cd00000-0cd3ffff : /soc/arm,smmu-mmss@cd00000
17817000-17817fff : msm-watchdog
17900000-1790dfff : msm-gladiator-erp
80000000-857fffff : System RAM
80080000-817fffff : Kernel code
82330000-82945fff : Kernel data
88f00000-8aafffff : System RAM
95300000-17e3bffff : System RAMCopy to clipboard
### /proc/vmallocinfo
This file shows detailed information about virtual address allocation through vmalloc or
ioremap.
The following is the sample of this file:
0xbf000000-0xbf002000 8192 module_alloc_update_bounds+0xc/0x5c pages=1 vmalloc
0xbf004000-0xbf008000 16384 module_alloc_update_bounds+0xc/0x5c pages=3 vmalloc
0xee800000-0xef800000 16777216 iotable_init+0x0/0xb0 phys=36800000 ioremap
0xf0000000-0xf0002000 8192 of_iomap+0x30/0x38 ioremap
0xf0002000-0xf0004000 8192 of_iomap+0x30/0x38 ioremap
0xf0004000-0xf000c000 32768 gen_pool_add_virt+0x48/0xb8 pages= 7 vmalloc
0xf000c000-0xf000e000 8192 msm_pm_setup_saved_state+0xcc/0x1bc ioremap
………………………
0xf0174000-0xf0176000 8192 msm_cpu_status_probe+0xd8/0x20c ioremap
0xf0f24000-0xf0f28000 16384 _kgsl_sharedmem_page_alloc+0xa0/0x41c pages=3 vmalloc
0xf0f39000-0xf0f3e000 20480 _kgsl_sharedmem_page_alloc+0xa0/0x41c pages=4 vmalloc
0xf0f61000-0xf0f66000 20480 _kgsl_sharedmem_page_alloc+0xa0/0x41c pages=4 vmalloc
……….
0xfa400000-0xfa600000 2097152 iotable_init+0x0/0xb0 phys=fa00000 ioremap
0xfa71e000-0xfa71f000 4096 iotable_init+0x0/0xb0 phys=f991e000 ioremap
0xfefd8000-0xff000000 163840 pcpu_get_vm_areas+0x0/0x56c vmallocCopy to clipboard
### /sys/kernel/debug/memblock interface
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
The memblock interface on the debugfs file system provides details about the available and
reserved memory regions in the system. The following files are available in this
interface:
- [/sys/kernel/debug/memblock/memory](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html#mtl1520725114834__section_N10015_N1000E_N10001)
- [/sys/kernel/debug/memblock/reserved](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html#mtl1520725114834__section_N10038_N1000E_N10001)
### /sys/kernel/debug/memblock/memory
This file provides details about all available memory regions (HLOS and non-HLOS) visible to the
Linux kernel. To know the overall memory accessible to the kernel, calculate the
difference between start and end addresses of each of the regions and add the values to
get the total occupied RAM. The remaining memory, which is the difference between the
RAM size of the device and the occupied RAM, is the non-HLOS memory or the memory
occupied by other subsystems.
To view the contents of this file, run the command:
cat sys/kernel/debug/memblock/memoryCopy to clipboard
Sample output:
0: 0x0000000080000000..0x00000000857fffff
1: 0x0000000088f00000..0x000000008aafffff
2: 0x0000000095300000..0x000000017e3bffff
Copy to clipboard
### /sys/kernel/debug/memblock/reserved
This file provides details about all reserved memory regions in the system.
To view the contents of this file, run the command:
cat sys/kernel/debug/memblock/reservedCopy to clipboard
Sample output:
0: 0x0000000080080000..0x0000000082944fff
1: 0x0000000083200000..0x0000000083259bb4
2: 0x0000000083400000..0x00000000839506c9
3: 0x00000000f5800000..0x00000000ffbfffff
4: 0x00000000ffff7000..0x00000000ffffefff
5: 0x00000000ffffff40..0x00000000ffffff77
6: 0x00000000ffffff80..0x00000000ffffffb7
7: 0x00000000ffffffc0..0x00000000fffffff7
8: 0x0000000179258000..0x000000017d9fffff
9: 0x000000017da17000..0x000000017da1ffff
10: 0x000000017da20e00..0x000000017da26fff
11: 0x000000017da27300..0x000000017da2735f
12: 0x000000017da27380..0x000000017da273df
13: 0x000000017da27400..0x000000017da2755f
14: 0x000000017da27580..0x000000017da27587
15: 0x000000017da275c0..0x000000017da275c7
16: 0x000000017da29600..0x000000017da29924
17: 0x000000017da29940..0x000000017da29c64
18: 0x000000017da29c80..0x000000017da29fa4
19: 0x000000017da29fac..0x000000017da2a3f8
20: 0x000000017da2a3fc..0x000000017da2a42e
21: 0x000000017da2a430..0x000000017da2a45e
22: 0x000000017da2a460..0x000000017e3bffffCopy to clipboard
### Memory leak
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
To debug kernel memory leak issues, enable the following configuration:
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE= 4000
CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=yCopy to clipboard
By default, a kernel thread scans the memory every 10 minutes and prints the number of
new unreferenced objects found. For
example,
unreferenced object 0xec26f000 (size 4096):
comm "Binder_2", pid 4592, jiffies 8848 (age 336.710s)
hex dump (first 32 bytes):
ec 4d f8 c0 02 00 00 00 00 00 00 00 00 00 00 00 .M..............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[] kmem_cache_alloc_trace+0x17c/0x238
[] ddl_client_transact+0xd0/0x158
[] ddl_open+0x4c/0x194
[] vcd_init_client_context+0x14/0x9c
[] vcd_open_in_ready+0x3c/0x94
[] vcd_open+0x214/0x274
[] vid_dec_open_client+0x1d0/0x288
[] vid_dec_open+0x30/0x7c
[] chrdev_open+0x10c/0x134
[] __dentry_open.isra.12+0x190/0x29c
[] do_last.isra.29+0x690/0x6c0
[] path_openat+0xb8/0x35c
[] do_filp_open+0x2c/0x78
[] do_sys_open+0xd8/0x170
[] ret_fast_syscall+0x0/0x30
[] 0xffffffffCopy to clipboard
The `KMEMleak` option can be disabled at boot time by passing the
`kmemleak=off` on the kernel command line.
For details about the kmemleak.txt file, see [https://www.kernel.org/doc/Documentation](https://www.kernel.org/doc/Documentation).
Following are the additional kernel configuration options available to track the
allocator of each page of memory:
- `CONFIG_PAGE_OWNER`
- `CONFIG_PAGE_OWNER_ENABLE_DEFAULT`
- `CONFIG_PAGE_EXTENSION`
Parsing all pages with these options enabled can help in case of multiple allocations,
which might be a memory leak issue.
### Memory corruption
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
To identify memory corruption issues, enable the following options in the kernel
configuration file:
- `CONFIG_PAGE_POISONING`
- `CONFIG_SLUB_DEBUG_ON`
- `CONFIG_DEBUG_LIST`
- `CONFIG_SLUB_DEBUG`
Sample log:
BUG :
-----------------------------------------------
INFO: -
INFO: Slab
INFO: Object
INFO: Allocated in age= cpu= pid=
INFO: Freed in age= cpu= pid=Copy to clipboard
For details about slub debugging, see Documentation/vm/slub.txt
available at [https://www.kernel.org/doc/Documentation](https://www.kernel.org/doc/Documentation).
### Out of memory
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
When the system cannot allocate a page, the kernel logs display a message such as the
following:
<4>[12146.861355] Thread-430: page allocation failure: order:0, mode:0x10d2
<4>[12146.951687] Mem-info:
<4>[12146.953909] Normal per-cpu:
<4>[12146.956686] CPU 0: hi: 186, btch: 31 usd: 61
<4>[12146.961489] CPU 1: hi: 186, btch: 31 usd: 0
<4>[12146.966235] HighMem per-cpu:
<4>[12146.969122] CPU 0: hi: 186, btch: 31 usd: 54
<4>[12146.973877] CPU 1: hi: 186, btch: 31 usd: 0
…….
<4>[12147.010770] Normal free:53192kB min:3508kB low:4384kB high:5260kB ……..
<4>[12147.050805] lowmem_reserve[]: 0 9022 9022
<4>[12147.054610] HighMem free:153360kB min:512kB low:1824kB high:3140kB ………
<4>[12147.095453] lowmem_reserve[]: 0 0 0
<4>[12147.098617] Normal: 118*4kB 232*8kB 161*16kB 110*32kB 34*64kB 9*128kB 10*256kB 6*512kB 7*1024kB 6*2048kB 4*4096kB = 53224kB
<4>[12147.115455] HighMem: 2774*4kB 11769*8kB 3005*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153360kB
………………Copy to clipboard
These messages indicate that the system could not allocate the requested page. The top line
in the log message provides several details about the out of memory issue. For example, in
the following log:
<4>[1214.855361] Thread-4: page allocation failure: order:2, mode:0x10d2Copy to clipboard
- **order**: indicates the size of the page; in this example, 2^2 x PAGE\_SIZE (4
K) = 16 K
- Linux uses a buddy allocator that allocates pages in powers of 2.
- The maximum size of the buddy allocator is the order of 10 = 2^10 x 4 kB
= 4 MB
- For allocation > 4 MB, use an alternate allocation method such as
contiguous memory allocator (CMA)
- For failure of higher-order allocations, see if the memory can be virtually
contiguous instead of physically contiguous.
- **mode**: indicates the type of page that was requested
- `mode` provides information about GFP flags. In this example,
`mode` is `0x10d2`.
- `mode` is the result of OR operation on all the GFP flags in
the allocation.
- Pages available in the system
A page allocation failure message prints details
about what was available in the system, for example:
Normal: 118*4kB 232*8kB 161*16kB 110*32kB 34*64kB 9*128kB 10*256kB 6*512kB 7*1024kB 6*2048kB 4*4096kB = 53224kB
HighMem: 2774*4kB 11769*8kB 3005*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153360kBCopy to clipboard
### IOMMU page fault
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
IOMMU is also known as the system MMU (SMMU). It performs memory management functions on
behalf of masters that do not have their own MMU. The IOMMU hardware block allows virtually
contiguous memory to be backed by physically non-contiguous pages. Memory translation logic
in the IOMMU is the same as the logic in the CPU MMU.
One of the most commonly seen IOMMU issues is the IOMMU page fault. IOMMU page faults
result when the page to be accessed is mapped in the page table but is not found in memory.
The fault handler gets the context bank instance of the IOMMU and dumps out the registers
for this context.
The following log indicates an IOMMU page fault:
[ 47.228992] msm_iommu_v1: Unexpected IOMMU page fault!
[ 47.233115] msm_iommu_v1: name = mdp_iommu
[ 47.237238] msm_iommu_v1: context = mdp_0 (0)
[ 47.241507] msm_iommu_v1: Interesting registers:
[ 47.246149] msm_iommu_v1: FAR = 0000000000000000
[ 47.250970] msm_iommu_v1: PAR = 0000000000000000
[ 47.255834] msm_iommu_v1: FSR = 00000002 [TF ]
[ 47.260540] msm_iommu_v1: FSYNR0 = 000005a1 FSYNR1 = 00030005
[ 47.266528] msm_iommu_v1: TTBR0 = 0000000071a28000
[ 47.271370] msm_iommu_v1: TTBR1 = 0000000000000000
[ 47.276248] msm_iommu_v1: SCTLR = 00001043 ACTLR = 70000000
[ 47.282221] msm_iommu_v1: CBAR = 00000000 CBFRSYNRA = 00000000
[ 47.288521] msm_iommu_v1: PRRR = ff0a81a8 NMRR = 40e040e0
[ 47.294461] msm_iommu_v1: NOTE: Value actually unknown for CBAR
[ 47.300394] msm_iommu_v1: NOTE: Value actually unknown for CBFRSYNRA
[ 47.306717] msm_iommu_v1: Page table in DDR shows PA = 0Copy to clipboard
The log message contains the following details:
- **name**: Hardware block that took the fault
- **FAR**: Fault Address Register, the address at which the fault occurred
- **FSR**: Fault Status Register
- Translation fault (TF)
- Access permission fault (APF)
- Stalled status (SS)
One of the most important registers in IOMMU debugging is the fault status register (FSR).
This register has read/write-clear access. The read operation on this register reads the
value in the register while the write operation clears the bits corresponding to 1s in the
written data, and leaves the bits corresponding to 0s unchanged. This prevents inadvertent
clearing of new faults when writing the register to clear an old fault. Some of the useful
bits in this register are:
- [Bit 1]: TF – Translation fault (invalid page table entry)
- [Bit 2]: AFF – Access fault
- [Bit 3]: APF – Permission fault (write to read only region, and so on)
- [Bit 4]: TLBMF – TLB miss fault
- [Bit 5]: HTWDEEF – Hardware table walk decode error external fault
- [Bit 6]: HTWSEEF – Hardware table walk slave error external fault
- [Bit 7]: MHF – Multiple hits in TLB
- [Bit 16]: SL – Second-level fault (fault occurred in second level of page
table)
- [Bit 30]: SS – Stalled status
- [Bit 31]: MULTI – Multiple faults
The flags TF or APF and SL indicate normal operation. Whereas TLBMF, HTW\*EEF, or MHF flags
indicate that there is an issue.
### IOMMU page table
The IOMMU page table dump provides a faulting address from the FAR and the register dump
in the kernel log. This faulting address is the virtual address, and the corresponding
physical address can be acquired from the page table. From the page table dump, the
address being accessed can be identified as mapped or not mapped.
Each IOMMU domain has a page table. The dump includes page tables for each of the
domains. There are currently six domains.
The following is an example of the domain 2 page table dump:
Domain: 2 [L2 cache redirect for page tables is OFF]
0x00000000--0x0001ffff [0x00020000] [UNMAPPED]
0x00020000--0x01807fff [0x017e8000] A:0x82a8e000--0x84275fff [0x017e8000] [R/W][4K]
0x01808000--0x01939fff [0x00132000] A:0xf0c24000--0xf0d55fff [0x00132000] [R/W][4K]
0x0193a000--0x0199ffff [0x00066000] A:0xf13fa000--0xf145ffff [0x00066000] [R/W][4K]
0x019a0000--0x01e85fff [0x004e6000] A:0xf966e000--0xf9b53fff [0x004e6000] [R/W][4K]
0x01e86000--0x01ffffff [0x0017a000] [UNMAPPED]
0x02000000--0x02feffff [0x00ff0000] A:0xf5a22000--0xf6a11fff [0x00ff0000] [R/W][4K]Copy to clipboard
In this example, the first column represents the virtual address, the second column
represents the number of bytes in the corresponding region of contiguous physical
addresses, and the third column refers to the physical addresses. The permissions are
also mentioned for each of these regions.
### Memory map
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
For information on the memory map, see the latest Release Notes.
### Stack corruption
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
Stack corruption occurs when memory locations in the stack are accessed due
to incorrect coding logic, which leads to a change in values at those memory locations.
Stack corruption can occur in the following ways:
- Due to bad code logic, all the stack memory is consumed, and memory is written beyond
the stack boundaries causing stack overflow.
- Accessing an array that is out of bounds.
- An undefined or freed pointer that points at a stack address.
- Corrupted return address of a caller function.
The Linux kernel provides several configurations that can help detect stack corruption. To
identify the stack corruption issues, enable the following kernel configuration options:
- `CONFIG_STACKPROTECTOR=Y`
- `CONFIG_STACKPROTECTOR_STRONG=Y`
The Kernel Address Sanitizer (KASAN) also helps in identifying some stack corruption
issues.
## Function tracer
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
Function tracer (ftrace) provides tracing utilities that do system-wide profiling and
tracing at runtime.
To use ftrace, enable the following config options:
- `CONFIG_FTRACE=y`
- `CONFIG_HAVE_FUNCTION_TRACER=y`
- `CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y`
- `CONFIG_HAVE_DYNAMIC_FTRACE=y`
- `CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y`
Following are some of the operations that can be performed using ftrace to debug kernel
issues.
### Dump ftrace information to kmsg buffer
To dump the ftrace information into the kmsg buffer anytime from the source code, call
the `ftrace_dump(DUMP_ALL)` function.
To increase the ftrace ring buffer size, run the command:
echo 200 > /sys/kernel/debug/tracing/buffer_size_kbCopy to clipboard
### Enable work queue trace
To enable work queue tracing, run the following commands:
mount -t debugfs nodev /sys/kernel/debugCopy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/workqueue/enableCopy to clipboard
echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_eventCopy to clipboard
cat /sys/kernel/debug/tracing/trace_pipeCopy to clipboard
cat /sys/kernel/debug/tracing/per_cpu/cpu1/traceCopy to clipboard
Sample output:
# tracer: nop
#
# entries-in-buffer/entries-written: 8682/8682 #P:1
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<...>-4783 [001] d.s4 7524.354249: workqueue_queue_work: work struct=f2d91ee4 function=free_css_set_work workqueue=f6427d80 req_cpu=1 cpu=1
-0 [001] d.h4 7524.424196: workqueue_queue_work: work struct=c10c32a8 function=def_work_fn workqueue=f55c7880 req_cpu=1 cpu=4
e.process.gapps-4758 [001] dNs4 7524.454227: workqueue_queue_work: work struct=c4727fa4 function=free_css_set_work workqueue=f6427d80 req_cpu=1 cpu=1
Binder_D-1693 [001] d.s3 7524.504198: workqueue_queue_work: work struct=cec6275c function=do_dbs_timer workqueue=f5424680 req_cpu=1 cpu=1
<...>-4832 [001] d.h3 7524.574194: workqueue_queue_work: work struct=c10c32a8 function=def_work_fn workqueue=f55c7880 req_cpu=1 cpu=4Copy to clipboard
### Enable IRQ trace
To enable interrupt tracing, run the following commands:
mount -t debugfs nodev /sys/kernel/debugCopy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/irq/irq_handler_entry/enableCopy to clipboard
cat /sys/kernel/debug/tracing/traceCopy to clipboard
cat /sys/kernel/debug/tracing/trace_pipeCopy to clipboard
Sample output:
TASK-PID CPU# TIMESTAMP FUNCTION
| | | | |
adbd-302 [000] 295.075568: irq_handler_entry: irq=132 name=msm_otg
adbd-302 [000] 295.075599: irq_handler_entry: irq=132 name=msm_hsusb
adbd-302 [000] 295.075782: irq_handler_entry: irq=132 name=msm_otg
adbd-302 [000] 295.075782: irq_handler_entry: irq=132 name=msm_hsusb
-0 [000] 295.076270: irq_handler_entry: irq=132 name=msm_otg
-0 [000] 295.076270: irq_handler_entry: irq=132 name=msm_hsusb
-0 [000] 295.077155: irq_handler_entry: irq=18 name=gp_timer
-0 [000] 295.087166: irq_handler_entry: irq=18 name=gp_timer
-0 [000] 295.097146: irq_handler_entry: irq=18 name=gp_timerCopy to clipboard
### Enable scheduler trace
To trace context switches between tasks, enable the `sched_switch` tracer
by running the following commands:
mount -t debugfs nodev /sys/kernel/debugCopy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enableCopy to clipboard
cat /sys/kernel/debug/tracing/traceCopy to clipboard
cat /sys/kernel/debug/tracing/trace_pipeCopy to clipboard
Sample
output:
TASK-PID CPU# TIMESTAMP FUNCTION
| | | | |
WindowManagerPo-484 [000] 994.583135: sched_switch: prev_comm=WindowManagerPo
SurfaceFlinger-334 [000] 994.583652: sched_switch: prev_comm=SurfaceFlinger
WindowManagerPo-484 [000] 994.584320: sched_switch: prev_comm=WindowManagerPo
GL updater-675 [000] 994.584370: sched_switch: prev_comm=GL updater
WindowManagerPo-484 [000] 994.584424: sched_switch: prev_comm=WindowManagerPoCopy to clipboard
### Find IRQ latency
To find maximum IRQ latency and pre-emption latency in the system, enable ftrace
configurations for `IRQOFF` and `PREEMPTIONOFF` as
following:
CONFIG_IRQSOFF_TRACER =Y
CONFIG_PREEMPT_TRACER =YCopy to clipboard
For this configuration to take effect, recompile the
kernel. This configuration helps to detect milliseconds latency effectively.
To enable tracing, run the following commands:
echo 0 > /sys/kernel/debug/tracing/tracing_enabledCopy to clipboard
echo 0 > /sys/kernel/debug/tracing/tracing_max_latencyCopy to clipboard
echo irqsoff > /sys/kernel/debug/tracing/current_tracerCopy to clipboard
echo 1 > /sys/kernel/debug/tracing/tracing_enabledCopy to clipboard
cat /sys/kernel/debug/tracing/traceCopy to clipboard
Configuring `tracing_max_latency = 0` shows the maximum latency observed
in the system. To detect any latency higher than the specified limit, set the threshold
level in microseconds. For example,
echo 2000 > /sys/kernel/debug/tracing/tracing_threshCopy to clipboard
Following example shows the traces for IRQ latency of 16 ms:
cat /sys/kernel/debug/tracing/trace
# tracer: irqsoff
#
# WARNING: FUNCTION TRACING IS CORRUPTED
# MAY BE MISSING FUNCTION EVENTS
# irqsoff latency trace v1.1.5 on 3.4.0-perf-g7736d93-dirty
# --------------------------------------------------------------------
# latency: 16757 us, #4/4, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:1)
# -----------------
# | task: EventThread-534 (uid:1000 nice:-9 policy:0 rt_prio:0)
# -----------------
# => started at: _raw_spin_lock_irqsave
# => ended at: _raw_spin_unlock_irqrestore
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
<...>-534 0d... 0us!: _raw_spin_lock_irqsave
<...>-534 0d..1 16756us+: _raw_spin_unlock_irqrestore
<...>-534 0d..1 16758us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore
<...>-534 0d..1 16780us :
=> trace_hardirqs_on
=> _raw_spin_unlock_irqrestore
=> clk_enable
=> mdss_dsi_clk_enable
=> mdss_dsi_clk_ctrl
=> mdss_dsi_clk_req
=> mdss_dsi_event_handler
=> mdss_mdp_ctl_intf_event
=> mdss_mdp_cmd_add_vsync_handler
=> mdss_mdp_overlay_vsync_ctrl
=> mdss_mdp_overlay_ioctl_handler
=> mdss_fb_ioctl
=> do_fb_ioctl
=> fb_ioctl
=> do_vfs_ioctl
=> sys_ioctl
=> ret_fast_syscall
/sys/kernel/debug/tracing #Copy to clipboard
For more information, see Documentation/trace/ftrace.txt available
at [https://www.kernel.org/doc/Documentation](https://www.kernel.org/doc/Documentation).
## Collect and parse RAM dumps
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
RAM dump is a snapshot of the system memory at the time of device failure and is useful
for debugging various crash issues.
### Enable RAM dump
The following kernel config is important for RAM dump feature and by default this
config option is enabled in the DEBUG and perf
build:
CONFIG_QCOM_SOC_DEBUG=yCopy to clipboard
By default, RAM dump is enabled in the DEBUG build but disabled in the perf/non-DEBUG
build using the kernel command-line parameter. To configure RAM dump, use the
`qcom_scm.download_mode` parameter in the
meta-qcom-hwe/conf/machine/qcm6490.conffile as:
- 0 – Disable
- 1 – Enable
For more information about the kernel configurations, see [Make kernel changes](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-3/yocto-kernel-support.html#make-kernel-changes).
### Collect RAM dump
To collect the RAM dump over USB, you can use the Product Configuration Assistant
Tool (PCAT). Download the PCAT tool from the [Qualcomm Package Manager](https://qpm.qualcomm.com/#/main/tools/find). After installing the PCAT, you can access the
PCAT user guide at the following locations:
- Windows host: C:\Program
Files(x86)\Qualcomm\PCAT\Docs
- Linux host: /opt/qcom/PCAT/Docs/
To collect RAM dumps using PCAT, do the following:
1. Launch PCAT on a Windows machine and connect the device, which is already in
the RAM dump mode.
If necessary, you can trigger a device crash by
clicking **CRASH** on the PCAT interface.
After the device is connected, the PCAT
automatically starts capturing the RAM dump and dump collection progress
is displayed on the PCAT UI.
2. To view the dumps, click **OPEN...**.
The following is an example
showing the contents of a RAM dump folder.
Note: Linux host uses a
command-line interface instead of the GUI for the PCAT-based RAM dump capture,
such as `PCAT -PLUGIN CC -DEVICE -DUMPDIR /tmp -RESET TRUE
-UNIQUETS TRUE`.
## Linux RAM dump parser
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
Linux RAM dump parser (RAMParser) is an open-source tool used to parse RAM dump on
Qualcomm Linux devices. The RAMParser processes the RAM dump using the Linux kernel
symbol file, which includes vmlinux, kernel object modules and extracts useful
information such as process stacks, IRQ, and workqueues.
Note: The RAMParser tool is built and verified only for
Windows.
### Prerequisites
- RAM dump and the corresponding vmlinux file
- Software prerequisites
- Windows machine
- Python 3.7+ or a later
version
python -m pip install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org prettytableCopy to clipboard
python -m pip install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org pyelftoolsCopy to clipboard
- Pyelftools package
1. Download Pyelftools package from [https://github.com/eliben/pyelftools](https://github.com/eliben/pyelftools).
2. Unzip the downloaded file and look for
pyelftools-master>elftools folder.
3. Copy elftools folder to <installed Python
path>\Lib\site-packages.
- RAMParser software
Download the RAMParser software from [https://git.codelinaro.org/clo/la/platform/vendor/qcom-opensource/tools/-/tree/opensource-tools.lnx.1.0](https://git.codelinaro.org/clo/la/platform/vendor/qcom-opensource/tools/-/tree/opensource-tools.lnx.1.0).
Note: The RAMParser software must be always present
in the C drive on a Windows machine.
- (Optional) TRACE32 software 2023.12 or a later version on C drive
(C:\T32)
The TRACE32 is used to load RAM dumps in the TRACE32
simulator using the RAMParser output files.
### Set up toolchains
The RAMParser requires access to gdb and nm. You can specify the paths to the gdb and
nm tools in one of the following ways:
- Using --gdb-path and --nm-path to specify the absolute path
- Using CROSS\_COMPILE to specify the prefix
- Using local\_settings.py file
Note: Just having gdb or nm on the path is not supported as
there are too many variations on the names to invoke.
To use the local\_settings.pyfile, do the following:
1. Create a directory named ramparser\_utils\utils in the
root folder.
2. Download the toolchain in the ramparser\_utils\utils
folder from [https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads](https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads).
### local\_settings.py
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
The RAMParser attempts to understand most of the settings automatically. However, there
are some settings that are unique to the runtime environment. These unique settings are
specified in the local\_settings.py file. As the
local\_settings.py is a python file, it can take advantage of
the python features.
Note: For RAMParser to pick the correct path
for toolchain utilities, add the local\_settings.py file to the
path <root>/tools/linux-ramdump-parser-v2
folder.
The format of the local\_settings.py file
is:
= Copy to clipboard
The local\_settings.pyfile supports the following features:
- gdb\_path: absolute path to the gdb tool for the RAM dumps
- nm\_path: absolute path to the nm tool for the RAM dumps
- gdb64\_path: absolute path to the 64‑bit gdb tool for the RAM dumps
- nm64\_path: absolute path to the 64‑bit nm tool for the RAM dumps
- qtf\_path: absolute path to qtf tool executable
### Example - local\_settings.py
import parser_util,os,sys
path = os.path.abspath(os.path.dirname(__file__))
ramparser_utils_path = os.path.abspath(os.path.join(path, "../../ramparser_utils"))
if parser_util.get_system_type() == 'Linux':
gdb_path = "/prj/l4linux/arm_compilers/gcc-linaro-arm-linux-gnueabihf-4.8-2013.06_linux/bin/arm-linux-gnueabihf-gdb"
nm_path = "/prj/l4linux/arm_compilers/gcc-linaro-arm-linux-gnueabihf-4.8-2013.06_linux/bin/arm-linux-gnueabihf-nm"
objdump_path = "/prj/l4linux/arm_compilers/gcc-linaro-arm-linux-gnueabihf-4.8-2013.06_linux/bin/arm-linux-gnueabihf-objdump"
gdb64_path = "/prj/l4linux/arm64-linaro-compiler/gcc-linaro-aarch64-linux-gnu-4.8-2013.09_linux/bin/aarch64-linux-gnu-gdb"
nm64_path = "/prj/l4linux/arm64-linaro-compiler/gcc-linaro-aarch64-linux-gnu-4.8-2013.09_linux/bin/aarch64-linux-gnu-nm"
objdump64_path = "/prj/l4linux/arm64-linaro-compiler/gcc-linaro-aarch64-linux-gnu-4.8-2013.09_linux/bin/aarch64-linux-gnu-objdump"
else:
#if not linux assume windows
nm_path = ramparser_utils_path + "\\utils\\arm-none-eabi-nm.exe"
gdb_path = ramparser_utils_path + "\\utils\\arm-none-eabi-gdb.exe"
objdump_path = ramparser_utils_path + "\\utils\\objdump.exe"
Copy to clipboard
### RAMParser commands
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
To parse dumps using RAMParser, run the following
command:
python ramparse.py --vmlinux --auto-dump --force-hardware --mod_path -o Copy to clipboard
- ``: Always use `qcm6490` for QCM6490,
QCS6490, and QCS5430 chipsets.
- python ramparse.py: invokes the RAMParser.
- `mod_path`: All the unstripped kernel modules must be copied into
a folder and can be referenced as the symbol path for
`--mod_path` option.
- ``: You can pass parser options in the command
to extract specific data from the RAM dump. Additionally, you can pass subparser
options in the command to extract the relevant data. For example, to extract
Ftrace information, pass `--dump-ftrace` along with the related
arguments, run the
command:
python \tools\linux-ramdump-parser-v2\ramparse.py --vmlinux --auto-dump --force-hardware --dump-ftrace --ftrace-args=rwmmio --ftrace_buffer_size_kb 4096 --mod_path -o Copy to clipboard
Similarly,
to extract the kconfig information, pass `--print-kconfig`. There
are multiple options available to parse the dumps and store the output in a
folder.
The following figure shows the available RAMParser commands.

### RAMParser logs
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
The RAMParser generates an extensive amount of data regarding workqueues, processes
states, call stacks. The following table lists the important files that are generated as
part of RAMParser execution:
| Filename | Description |
| --- | --- |
| dmesg\_TZ.txt | Contains kernel logs, runqueues, workqueues, IRQ statistics. |
| mem\_stat.txt, memory.txt | Provides system memory available/free statistics. |
| tasks.txt | Provides kernel space call stack for all processes. |
| devicetree.dtb | Device tree blob used by the kernel. |
| launch\_t32.bat | TRACE32 simulator launcher file for loading RAM dumps. |
| timerlist.txt | Provides the list of active timers. |
The following screenshots show sample data generated by the RAMParser.
**Runqueue (dmesg\_TZ.txt)**

**IRQ state (dmesg\_TZ.txt)**

**Sample kernel crash (dmesg\_TZ.txt)**

**Timer list of cores (timerlist.txt)**

**Workqueue (dmesg\_TZ.txt)**

**Memory statistics (mem\_stat.txt)**

**CPU frequency (dmesg\_TZ.txt)**

**TRACE32 simulator launcher (launch\_t32.bat)**

## Parse RAM dumps using QCAP
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
### About this task
Qualcomm Crash Analysis Portal (QCAP) is a powerful tool to parse logs from all
subsystems and determine on which subsystem the crash occurred first.
This tool is currently available for users who have full access to the proprietary
software shipped with Qualcomm Linux. For information on how to parse subsystem
dumps using QCAP, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70014-12A/debugging_linux_kernel.html#parse_ram_dumps_using_qcap).
## Crash utility
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
Crash utility is an open-source tool to debug kernel using a gdb based command-line
interface over RAM dumps.
**Prerequisites:**
- `kaslr_offset` and `kimage_voffset` values
By
default, the Kernel Address Space Layout Randomization (KASLR) feature is
enabled on the Qualcomm Linux build. For crash utility to work on KASLR
enabled kernel, the `kaslr_offset` and
`kimage_voffset` information is required, which can be
extracted from RAMparser output file,
dmesg\_TZ.txt.
Following is an excerpt from the
sample RAMparser output file, dmesg\_TZ.txt, providing
the `kaslr_offset` and `kimage_voffset`
values.
The kaslr_offset extracted is: 0x340c3d320000
...
The kimage_voffset extracted is: ffffb40bbf600000Copy to clipboard
- Base address of the kernel binaries (`DDRCS*.BIN`)
When RAM
dump is captured using PCAT, dump\_info.txt and load.cmm files are captured
in addition to the RAM dump. These files contain the base address of the
kernel binaries (`DDRCS*.BIN`).
**Download and build crash utility on Linux host**
To download and build the crash utility, run the following
commands:
git clone https://github.com/crash-utility/crashCopy to clipboard
make target=ARM64Copy to clipboard
make extensions=ARM64Copy to clipboard
For more information, see [crash/README at master · crash-utility/crash · GitHub](https://github.com/crash-utility/crash/blob/master/README).
**Run crash utility**
The following is the sample command to run the crash utility on an Ubuntu host:
./crash /vmlinux /DDRCS0_0.BIN@0x80000000,/DDRCS1_0.BIN@0x100000000,/DDRCS1_1.BIN@0x180000000 -m vabits_actual=48 -m max_physmem_bits=48 -m kimage_voffset=0xffffb40bbf600000 --kaslr=0x340c3d320000Copy to clipboard
- Replace `` with the vmlinux RAM dump path.
- `kimage_voffset`, `kaslr` are extracted from the
RAMparser output file, dmesg\_TZ.txt.
- DDR offsets like `0x80000000` are extracted from the
dump\_info.txt file available in the RAM dump collected
using PCAT.
Sample output:
crash 8.0.4
Copyright (C) 2002-2022 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
NOTE: setting vabits_actual to: 48
NOTE: setting max_physmem_bits to: 48
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=aarch64-elf-linux".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
WARNING: cpu 0: cannot find NT_PRSTATUS note
WARNING: cpu 1: cannot find NT_PRSTATUS note
WARNING: cpu 2: cannot find NT_PRSTATUS note
WARNING: cpu 3: cannot find NT_PRSTATUS note
WARNING: cpu 4: cannot find NT_PRSTATUS note
WARNING: cpu 5: cannot find NT_PRSTATUS note
WARNING: cpu 6: cannot find NT_PRSTATUS note
WARNING: cpu 7: cannot find NT_PRSTATUS note
KERNEL: /test/vmlinux [TAINTED]
DUMPFILES: /var/tmp/ramdump_elf_PBaLfj [temporary ELF header]
/test/DDRCS0_0.BIN
/test/DDRCS1_0.BIN
/test/DDRCS1_1.BIN
CPUS: 8 [OFFLINE: 7]
DATE: Sun Jan 6 05:30:34 +0530 1980
UPTIME: 00:00:48
LOAD AVERAGE: 1.60, 0.51, 0.18
TASKS: 532
NODENAME: qcm6490
RELEASE: 6.6.17-debug
VERSION: #1 SMP PREEMPT Mon Mar 25 04:52:52 UTC 2024
MACHINE: aarch64 (unknown Mhz)
MEMORY: 5.5 GB
PANIC: "Kernel panic - not syncing: sysrq triggered crash"
PID: 1419
COMMAND: "sh"
TASK: ffff1756f61ea000 [THREAD_INFO: ffff1756f61ea000]
CPU: 0
STATE: TASK_RUNNING (PANIC)Copy to clipboard
crash> manCopy to clipboard
* files mod sbitmapq union
alias foreach mount search vm
ascii fuser net set vtop
bpf gdb p sig waitq
bt help ps struct whatis
btop ipcs pte swap wr
dev irq ptob sym q
dis kmem ptov sys
eval list rd task
exit log repeat timer
extend mach runq tree
crash version: 8.0.4 gdb version: 10.2
For help on any command above, enter "help ".
For help on input options, enter "help input".
For help on output options, enter "help output".Copy to clipboard
crash> logCopy to clipboard
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
[ 0.000000] Linux version 6.6.17-debug (oe-user@oe-host) (aarch64-qcom-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Mon Mar 25 04:52:52 UTC 2024
[ 0.000000] KASLR enabled
[ 0.000000] Machine model: Qualcomm Technologies, Inc. Robotics RB3gen2 addons vision mezz platform
[ 0.000000] efi: EFI v2.7 by Qualcomm Technologies, Inc.
[ 0.000000] efi: MEMATTR=0x9ccf6018 INITRD=0x9cceaf18 RNG=0x9cce3018 MEMRESERVE=0x9ccea818
[ 0.000000] random: crng init done
[ 0.000000] Reserved memory: created CMA memory pool at 0x00000000fd000000, size 12 MiB
[ 0.000000] OF: reserved mem: initialized node adsp-heap, compatible id shared-dma-poolCopy to clipboard
crash> p memdumpCopy to clipboard
memdump = $1 = {
table_phys = 4110417920,
table = 0xffff80008087d000
}Copy to clipboard
For more information, see the following:
- [https://crash-utility.github.io/](https://crash-utility.github.io/)
- [https://man7.org/linux/man-pages/man8/crash.8.html](https://man7.org/linux/man-pages/man8/crash.8.html)
## Subsystem dumps
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70014-12/topic/debugging_linux_kernel.html)
This feature is currently available for users who have full access to the proprietary
software shipped with Qualcomm Linux. For information on how to enable and capture core
dumps of subsystems, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70014-12A/debugging_linux_kernel.html#subsystem_ram_dumps).
Last Published: Jul 12, 2024
[Previous Topic
Debug Linux user space](https://docs.qualcomm.com/bundle/publicresource/80-70014-12/topics/using_open_source_debug_tools.md) [Next Topic
Debug common system issues](https://docs.qualcomm.com/bundle/publicresource/80-70014-12/topics/general_system_debugging.md)