# Debug Linux kernel space issues
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
At a high-level, issues in the kernel space are categorized as kernel panic and bugs. The
following figure illustrates the logs, command-line options, and dumps that can be used
to debug issues in the kernel space.
Figure : Resources to debug issues in kernel space
Page-1 Dynamic debug /proc file system RAM dump RAM parser Logs Command line Kernel debugging configuration configuration Dumps Function tracer Debugfs file system Crash utility
Waiting for more than one second on a spinlock Freeing an already freed lock Reinitializing a lock that was already used ] (unwind_backtrace+0x0/0x138) from [] (warn_slowpath_common+0x4c/0x64)
<6> [] (warn_slowpath_common+0x4c/0x64) from [] (warn_slowpath_fmt+0x30/0x40)
<6> [] (warn_slowpath_fmt+0x30/0x40) from [] (__list_del_entry+0xa0/0xd0)
<6> [] (__list_del_entry+0xa0/0xd0) from [] (account_entity_dequeue+0x84/0x94)
<6> [] (account_entity_dequeue+0x84/0x94) from [] (dequeue_task_fair+0x64/0x190)Copy to clipboard
**Sample crash signature 2**
<4>WARNING: at kernel/lib/list_debug.c:52 __list_del_entry+0x8c/0xac()
<4> list_del corruption, e240c500->prev is LIST_POISON2 (00200200)
<4> Modules linked in: wlan(PO) cfg80211 adsp_loader
<4> [] (unwind_backtrace+0x0/0x11c)
from [] (warn_slowpath_common+0x4c/0x64)
<4> [] (warn_slowpath_common+0x4c/0x64)
from [] (warn_slowpath_fmt+0x2c/0x3c)
<4> [] (warn_slowpath_fmt+0x2c/0x3c)
from [] (__list_del_entry+0x8c/0xac)
<4> [] (__list_del_entry+0x8c/0xac)
from [] (list_del+0xc/0x24)
<4> [] (list_del+0xc/0x24)
from [] (binder_thread_read+0x488/0xb70)
<4> [] (binder_thread_read+0x488/0xb70)
from [] (binder_ioctl+0x220/0x5b8)Copy to clipboard
**Sample crash signature 3**
<4> WARNING: at kernel/lib/list_debug.c:47 __list_del_entry+0x90/0xb0()
<4> list_del corruption, d6ed3720->next is LIST_POISON1 (00100100)
<4> Modules linked in:
<4> [] (unwind_backtrace+0x0/0x12c) from [] (warn_slowpath_common+0x4c/0x64)
<4> [] (warn_slowpath_common+0x4c/0x64) from [] (warn_slowpath_fmt+0x2c/0x3c)
<4> [] (warn_slowpath_fmt+0x2c/0x3c) from [] (__list_del_entry+0x90/0xb0)
<4> [] (__list_del_entry+0x90/0xb0) from [] (list_del+0xc/0x28)
<4> [] (list_del+0xc/0x28) from [] (bam_mux_write_done+0x34/0x11c)Copy to clipboard
### Spinlock corruption issues
When `CONFIG_DEBUG_SPINLOCK` and `CONFIG_DEBUG_MUTEXES`
kernel configuration options are enabled, the following crash signatures indicating
spinlock corruption issues can be identified:
**Crash signature 1**
<0>BUG: spinlock lockup on CPU#1, ndroid.launcher/1071
<0> lock: 0xd5e8f480, .magic: dead4ead, .owner: /-1, .owner_cpu: -1
<4> [] (spin_dump+0x74/0x84) from []
(do_raw_spin_lock+0x144/0x188)
<4> [] (do_raw_spin_lock+0x144/0x188) from [] (kgsl_mmu_pt_get_flags+0x18/0x44)
<4> [] (kgsl_mmu_pt_get_flags+0x18/0x44) from [] (adreno_ringbuffer_submitcmd+0x168/0x228)
<4> [] (adreno_ringbuffer_submitcmd+0x168/0x228) from [] (sendcmd+0x3c/0x254)Copy to clipboard
**Crash signature 2**
<0> BUG: spinlock lockup on CPU#2, kworker/2:0H/2910
<0> lock: kpss_clock_reg_lock+0x0/0x10, .magic: dead4ead, .owner:
kworker/3:0H/2904, .owner_cpu: 3
<6> kworker/2:0H (2910): undefined instruction: pc=c0963098
kernel BUG at kernel/lib/spinlock_debug.c:95!
<4> [] (spin_dump+0x7c/0x94) from []
(do_raw_spin_lock+0xcc/0x164)
<4> [] (do_raw_spin_lock+0xcc/0x164) from [] (_raw_spin_lock_irqsave+0x20/0x28)
<4> [] (_raw_spin_lock_irqsave+0x20/0x28) from [] (__kpss_mux_set_sel+0x14/0x80)
<4> [] (__kpss_mux_set_sel+0x14/0x80) from [] (kpss_mux_set_sel+0x18/0x20)Copy to clipboard
### Slub poisoning issues
When `CONFIG_SLUB` and `CONFIG_SLUB_DEBUG` kernel
configuration options are enabled, the following crash signature that indicates a slub
poisoning issue can be identified:
<3>[ 3438.930472] =============================================================================
<3>[ 3438.937628] BUG kmalloc-64 (Tainted: G W O): Poison overwritten
<3>[ 3438.944223] -----------------------------------------------------------------------------
<3>[ 3438.944228]
<3>[ 3438.953861] INFO: 0xce308408-0xce30840b. First byte 0x0 instead of 0x6b
<3>[ 3438.960470] INFO: Allocated in kgsl_ioctl_drawctxt_create+0x2c/0x2ac age=4426 cpu=0 pid=21702
<3>[ 3438.968970] __slab_alloc.isra.37.constprop.43+0x4d4/0x534
<3>[ 3438.974435] kmem_cache_alloc_trace+0x240/0x258
<3>[ 3438.978947] kgsl_ioctl_drawctxt_create+0x2c/0x2ac
<3>[ 3438.983721] kgsl_ioctl+0xfc/0x324
<3>[ 3438.987108] do_vfs_ioctl+0x80/0x54c
<3>[ 3438.990669] sys_ioctl+0x38/0x5c
<3>[ 3438.993880] ret_fast_syscall+0x0/0x30
<3>[ 3438.997615] INFO: Freed in kgsl_release+0xb8/0xc0 age=10 cpu=3 pid=21842
<3>[ 3439.004298] __slab_free+0x30/0x308
<3>[ 3439.007770] kgsl_release+0xb8/0xc0
<3>[ 3439.011242] fput+0xcc/0x23c
<3>[ 3439.014106] filp_close+0x68/0x80
<3>[ 3439.017407] put_files_struct+0xd8/0x110
<3>[ 3439.021312] do_exit+0x164/0x860
<3>[ 3439.024524] do_group_exit+0x3c/0xb0
<3>[ 3439.028084] get_signal_to_deliver+0x2c4/0x59c
<3>[ 3439.032510] do_signal+0x90/0x480
<3>[ 3439.035809] do_notify_resume+0x50/0x5c
<3>[ 3439.039629] work_pending+0x24/0x28
<3>[ 3439.043101] INFO: Slab 0xc0fbcd20 objects=16 used=16 fp=0x (null) flags=0x0080
<3>[ 3439.050401] INFO: Object 0xce308400 @offset=1024 fp=0xce308900
<3>[ 3439.050405]
<3>[ 3439.057694] Bytes b4 ce3083f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
<3>[ 3439.066457] Object ce308400: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b kkkkkkkk....kkkk
<3>[ 3439.075051] Object ce308410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
<3>[ 3439.083639] Object ce308420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
<3>[ 3439.092234] Object ce308430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
<3>[ 3439.100828] Redzone ce308440: bb bb bb bb ....
<3>[ 3439.108463] Padding ce3084e8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
<3>[ 3439.117145] Padding ce3084f8: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
<4>[ 3439.125144] [] (unwind_backtrace+0x0/0x138) from [] (check_bytes_and_report+0xc0/0xe4)
<4>[ 3439.134771] [] (check_bytes_and_report+0xc0/0xe4) from [] (check_object+0x1b8/0x214)
<4>[ 3439.144237] [] (check_object+0x1b8/0x214) from [] (alloc_debug_processing+0x7c/0x150)
<4>[ 3439.153782] [] (alloc_debug_processing+0x7c/0x150) from [] (__slab_alloc.isra.37.constprop.43+0x4d4/0x534)
<4>[ 3439.165153] [] (__slab_alloc.isra.37.constprop.43+0x4d4/0x534) from [] (kmem_cache_alloc_trace+0x240/0x258)
<4>[ 3439.176611] [] (kmem_cache_alloc_trace+0x240/0x258) from [] (__get_vm_area_node.isra.26+0x84/0x174)
<4>[ 3439.187376] [] (__get_vm_area_node.isra.26+0x84/0x174) from [] (get_vm_area_caller+0x44/0x4c)
<4>[ 3439.197615] [] (get_vm_area_caller+0x44/0x4c) from [] (vmap+0x50/0x90)
<4>[ 3439.205866] [] (vmap+0x50/0x90) from [] (_kgsl_sharedmem_page_alloc+0x238/0x3d4)
<4>[ 3439.214979] [] (_kgsl_sharedmem_page_alloc+0x238/0x3d4) from [] (_gpumem_alloc+0xb0/0xfc)
<4>[ 3439.224874] [] (_gpumem_alloc+0xb0/0xfc) from [] (kgsl_ioctl_gpumem_alloc_id+0x40/0x1a8)
<4>[ 3439.234684] [] (kgsl_ioctl_gpumem_alloc_id+0x40/0x1a8) from [] (kgsl_ioctl+0xfc/0x324)
<4>[ 3439.244319] [] (kgsl_ioctl+0xfc/0x324) from [] (do_vfs_ioctl+0x80/0x54c)
<4>[ 3439.252738] [] (do_vfs_ioctl+0x80/0x54c) from [] (sys_ioctl+0x38/0x5c)
<4>[ 3439.260986] [] (sys_ioctl+0x38/0x5c) from [] (ret_fast_syscall+0x0/0x30)
<3>[ 3439.269400] FIX kmalloc-64: Restoring 0xce308408-0xce30840b=0x6bCopy to clipboard
### Page poisoning issues
When `CONFIG_DEBUG_PAGEALLOC` and `CONFIG_PAGE_POISONING`
kernel configuration options are enabled, the following crash signature that indicates a
page poisoning issue can be identified:
<1> Unable to handle kernel paging request at virtual address aaaaaaae
<1> pgd = e98b4000
<1> [aaaaaaae] *pgd=00000000
<0> Internal error: Oops: 5 [#1] PREEMPT SMP ARM
<4> Modules linked in: adsp_loader exfat_fs(P) exfat_core(P)
<4> CPU: 1 Tainted: P W (3.4.0-628250-eng #1)
<4> PC is at pid_nr_ns+0xc/0x3c
<4> LR is at do_task_stat+0x248/0x83c
<4> pc : [] lr : [] psr: a0000093Copy to clipboard
## CPU parameters
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
Various CPU parameters such as core frequency, CPU governor, and cpuidle states help you to
know more about the system and tune it accordingly.
For information about CPU parameters, see [https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-devices-system-cpu](https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-devices-system-cpu).
## Memory
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
Linux uses a virtual memory system. Therefore, the addresses that the user program accesses
do not correspond to the physical addresses that the hardware uses directly. Virtual memory
introduces a layer of indirection that allows programs running on the system to allocate
additional memory, apart from the physically available memory.
Memory management implementation covers the following areas:
- Management of physical pages in the memory
- Buddy system to allocate memory in large chunks
- Slab, slub, and slob allocators to allocate smaller chunks of memory
- vmalloc mechanism to allocate noncontiguous blocks of memory
- Address space of the processes
### /proc file system
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
The following files are available on the `/proc`file system:
- [/proc/meminfo](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html#zbq1520725114522__section_N10011_N1000E_N10001)
- [/proc/vmstat](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html#zbq1520725114522__section_N1002B_N1000E_N10001)
- [/proc/iomem](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html#zbq1520725114522__section_N10045_N1000E_N10001)
- [/proc/vmallocinfo](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html#zbq1520725114522__section_N10053_N1000E_N10001)
### /proc/meminfo
This file provides information about the distribution and usage of memory. To view the
contents of this file, run the following command:
cat /proc/meminfoCopy to clipboard
Sample output:
MemTotal: 3813532 kB
MemFree: 624836 kB
MemAvailable: 2098008 kB
Buffers: 40416 kB
Cached: 1484320 kB
SwapCached: 0 kB
Active: 1334816 kB
.
.
.
.
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2431048 kB
Committed_AS: 99995284 kB
VmallocTotal: 258867136 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
CmaTotal: 163840 kB
CmaFree: 1368 kBCopy to clipboard
For details about the parameters available in this file, see [https://docs.kernel.org/filesystems/proc.html](https://docs.kernel.org/filesystems/proc.html).
### /proc/vmstat
This file shows detailed virtual memory statistics from the kernel. Most of the
statistics are available only if the `CONFIG_VM_EVENT_COUNTERS` option in
the init/Kconfig file is enabled.
To view the contents of this file, run the following command:
cat /proc/vmstatCopy to clipboard
Sample output:
nr_free_pages 156290
nr_alloc_batch 132
nr_inactive_anon 108
nr_active_anon 165006
nr_inactive_file 212275
nr_active_file 168709
nr_unevictable 64
nr_mlock 64
nr_anon_pages 164982
nr_mapped 90366
nr_file_pages 381184:
:
unevictable_pgs_mlocked 0
unevictable_pgs_munlocked 0
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 0Copy to clipboard
For details about the parameters available in this file, see [https://man7.org/linux/man-pages/man8/vmstat.8.html](https://man7.org/linux/man-pages/man8/vmstat.8.html).
### /proc/iomem
This file shows the memory map of the system for its various device drivers. To view the
contents of this file, run the following command:
cat /proc/iomemCopy to clipboard
Sample output:
007781b8-007791b7 : vmpm
010aa000-010abfff : tsens_physical
010ac000-010ac003 : pshold-base
010ad000-010aefff : tsens_physical
01680000-0168ffff : /soc/arm,smmu-anoc1@1680000
016c0000-016fffff : /soc/arm,smmu-anoc2@16c0000
01d0101c-01d0101f : sp2soc_irq_status
01d01024-01d01027 : sp2soc_irq_clr
01d01028-01d0102b : sp2soc_irq_mask
.
.
.
0caa0000-0caa3fff : jpeg_hw
0caa4000-0caa47ff : fd_core
0caa5000-0caa53ff : fd_misc
0cd00000-0cd3ffff : /soc/arm,smmu-mmss@cd00000
17817000-17817fff : msm-watchdog
17900000-1790dfff : msm-gladiator-erp
80000000-857fffff : System RAM
80080000-817fffff : Kernel code
82330000-82945fff : Kernel data
88f00000-8aafffff : System RAM
95300000-17e3bffff : System RAMCopy to clipboard
### /proc/vmallocinfo
This file shows detailed information about virtual address allocation through
`vmalloc` or `ioremap`. To view the contents of this
file, run the following command:
cat /proc/vmallocinfoCopy to clipboard
Sample output:
0xbf000000-0xbf002000 8192 module_alloc_update_bounds+0xc/0x5c pages=1 vmalloc
0xbf004000-0xbf008000 16384 module_alloc_update_bounds+0xc/0x5c pages=3 vmalloc
0xee800000-0xef800000 16777216 iotable_init+0x0/0xb0 phys=36800000 ioremap
0xf0000000-0xf0002000 8192 of_iomap+0x30/0x38 ioremap
0xf0002000-0xf0004000 8192 of_iomap+0x30/0x38 ioremap
0xf0004000-0xf000c000 32768 gen_pool_add_virt+0x48/0xb8 pages= 7 vmalloc
0xf000c000-0xf000e000 8192 msm_pm_setup_saved_state+0xcc/0x1bc ioremap
………………………
0xf0174000-0xf0176000 8192 msm_cpu_status_probe+0xd8/0x20c ioremap
0xf0f24000-0xf0f28000 16384 _kgsl_sharedmem_page_alloc+0xa0/0x41c pages=3 vmalloc
0xf0f39000-0xf0f3e000 20480 _kgsl_sharedmem_page_alloc+0xa0/0x41c pages=4 vmalloc
0xf0f61000-0xf0f66000 20480 _kgsl_sharedmem_page_alloc+0xa0/0x41c pages=4 vmalloc
……….
0xfa400000-0xfa600000 2097152 iotable_init+0x0/0xb0 phys=fa00000 ioremap
0xfa71e000-0xfa71f000 4096 iotable_init+0x0/0xb0 phys=f991e000 ioremap
0xfefd8000-0xff000000 163840 pcpu_get_vm_areas+0x0/0x56c vmallocCopy to clipboard
### memblock interface
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
The `memblock`interface on the debugfs file system provides details about
the available and reserved memory regions in the system. The following files are available
in this interface:
- [/sys/kernel/debug/memblock/memory](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html#mtl1520725114834__section_N10015_N1000E_N10001)
- [/sys/kernel/debug/memblock/reserved](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html#mtl1520725114834__section_N10038_N1000E_N10001)
### /sys/kernel/debug/memblock/memory
This file provides details about all the available memory regions (HLOS and non-HLOS) visible to
the Linux kernel. To know the overall memory accessible to the kernel, calculate the
difference between start and end addresses of each of the regions and add the values to
get the total occupied RAM. The remaining memory, which is the difference between the
RAM size of the device and the occupied RAM, is the non-HLOS memory or the memory
occupied by other subsystems.
To view the contents of this file, run the following command:
cat /sys/kernel/debug/memblock/memoryCopy to clipboard
Sample output:
0: 0x0000000080000000..0x00000000857fffff
1: 0x0000000088f00000..0x000000008aafffff
2: 0x0000000095300000..0x000000017e3bffffCopy to clipboard
### /sys/kernel/debug/memblock/reserved
This file provides details about all reserved memory regions in the system.
To view the contents of this file, run the following command:
cat sys/kernel/debug/memblock/reservedCopy to clipboard
Sample output:
0: 0x0000000080080000..0x0000000082944fff
1: 0x0000000083200000..0x0000000083259bb4
2: 0x0000000083400000..0x00000000839506c9
3: 0x00000000f5800000..0x00000000ffbfffff
4: 0x00000000ffff7000..0x00000000ffffefff
5: 0x00000000ffffff40..0x00000000ffffff77
6: 0x00000000ffffff80..0x00000000ffffffb7
7: 0x00000000ffffffc0..0x00000000fffffff7
8: 0x0000000179258000..0x000000017d9fffff
9: 0x000000017da17000..0x000000017da1ffff
10: 0x000000017da20e00..0x000000017da26fff
11: 0x000000017da27300..0x000000017da2735f
12: 0x000000017da27380..0x000000017da273df
13: 0x000000017da27400..0x000000017da2755f
14: 0x000000017da27580..0x000000017da27587
15: 0x000000017da275c0..0x000000017da275c7
16: 0x000000017da29600..0x000000017da29924
17: 0x000000017da29940..0x000000017da29c64
18: 0x000000017da29c80..0x000000017da29fa4
19: 0x000000017da29fac..0x000000017da2a3f8
20: 0x000000017da2a3fc..0x000000017da2a42e
21: 0x000000017da2a430..0x000000017da2a45e
22: 0x000000017da2a460..0x000000017e3bffffCopy to clipboard
### Memory leak
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
To debug kernel memory leak issues, enable the following configuration options:
- `CONFIG_DEBUG_KMEMLEAK=y`
- `CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE= 4000`
- `CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y`
By default, a kernel thread scans the memory every 10 minutes and prints the number of
new unreferenced objects found. For example,
unreferenced object 0xec26f000 (size 4096):
comm "Binder_2", pid 4592, jiffies 8848 (age 336.710s)
hex dump (first 32 bytes):
ec 4d f8 c0 02 00 00 00 00 00 00 00 00 00 00 00 .M..............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[] kmem_cache_alloc_trace+0x17c/0x238
[] ddl_client_transact+0xd0/0x158
[] ddl_open+0x4c/0x194
[] vcd_init_client_context+0x14/0x9c
[] vcd_open_in_ready+0x3c/0x94
[] vcd_open+0x214/0x274
[] vid_dec_open_client+0x1d0/0x288
[] vid_dec_open+0x30/0x7c
[] chrdev_open+0x10c/0x134
[] __dentry_open.isra.12+0x190/0x29c
[] do_last.isra.29+0x690/0x6c0
[] path_openat+0xb8/0x35c
[] do_filp_open+0x2c/0x78
[] do_sys_open+0xd8/0x170
[] ret_fast_syscall+0x0/0x30
[] 0xffffffffCopy to clipboard
The `KMEMLEAK` option can be disabled at boot time by passing
`KMEMLEAK=off` on the kernel command line.
For details about the kmemleak.txt file, see [https://www.kernel.org/doc/Documentation](https://www.kernel.org/doc/Documentation).
The following are the additional kernel configuration options available to track the
allocator of each page of memory:
- `CONFIG_PAGE_OWNER`
- `CONFIG_PAGE_OWNER_ENABLE_DEFAULT`
- `CONFIG_PAGE_EXTENSION`
Parsing all pages with these options enabled can help when there are multiple
allocations, which might be a memory leak issue.
### Memory corruption
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
To identify memory corruption issues, enable the following kernel configuration options:
- `CONFIG_PAGE_POISONING`
- `CONFIG_SLUB_DEBUG_ON`
- `CONFIG_DEBUG_LIST`
- `CONFIG_SLUB_DEBUG`
Sample log:
BUG :
-----------------------------------------------
INFO: -
INFO: Slab
INFO: Object
INFO: Allocated in age= cpu= pid=
INFO: Freed in age= cpu= pid=Copy to clipboard
For more information on slub debugging, see Documentation/vm/slub.txt
available at [https://www.kernel.org/doc/Documentation](https://www.kernel.org/doc/Documentation).
### Out of memory
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
When the system cannot allocate a page, the kernel logs display a message such as the
following:
<4>[12146.861355] Thread-430: page allocation failure: order:0, mode:0x10d2
<4>[12146.951687] Mem-info:
<4>[12146.953909] Normal per-cpu:
<4>[12146.956686] CPU 0: hi: 186, btch: 31 usd: 61
<4>[12146.961489] CPU 1: hi: 186, btch: 31 usd: 0
<4>[12146.966235] HighMem per-cpu:
<4>[12146.969122] CPU 0: hi: 186, btch: 31 usd: 54
<4>[12146.973877] CPU 1: hi: 186, btch: 31 usd: 0
…….
<4>[12147.010770] Normal free:53192kB min:3508kB low:4384kB high:5260kB ……..
<4>[12147.050805] lowmem_reserve[]: 0 9022 9022
<4>[12147.054610] HighMem free:153360kB min:512kB low:1824kB high:3140kB ………
<4>[12147.095453] lowmem_reserve[]: 0 0 0
<4>[12147.098617] Normal: 118*4kB 232*8kB 161*16kB 110*32kB 34*64kB 9*128kB 10*256kB 6*512kB 7*1024kB 6*2048kB 4*4096kB = 53224kB
<4>[12147.115455] HighMem: 2774*4kB 11769*8kB 3005*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153360kB
………………Copy to clipboard
These messages indicate that the system could not allocate the requested page. The top line
in the log message provides several details about the out-of-memory issue. For example, in
the following log:
<4>[1214.855361] Thread-4: page allocation failure: order:2, mode:0x10d2Copy to clipboard
- **order**: Indicates the size of the page; in this example, 2^2 x PAGE\_SIZE
(4 K) = 16 K
- Linux uses a buddy allocator that allocates pages in powers of 2.
- The maximum size of the buddy allocator is the order of
10 = 2^10 x 4 kB = 4 MB.
- For allocation > 4 MB, use an alternate allocation method such as
contiguous memory allocator (CMA).
- For failure of higher-order allocations, examine whether the memory can be
virtually contiguous, instead of being physically contiguous.
- **mode**: Indicates the type of page that was requested
- `mode` provides information about get free pages (GFP) flags.
In this example, `mode` is `0x10d2`.
- `mode` is the result of OR operation on all the GFP flags in
the allocation.
- Pages available in the system
A page allocation failure message prints details
about the size of pages that were available in the system. For example:
Normal: 118*4kB 232*8kB 161*16kB 110*32kB 34*64kB 9*128kB 10*256kB 6*512kB 7*1024kB 6*2048kB 4*4096kB = 53224kB
HighMem: 2774*4kB 11769*8kB 3005*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153360kBCopy to clipboard
### IOMMU page fault
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
IOMMU is also known as the system MMU (SMMU). It performs memory management functions on
behalf of subsystems that do not have their own MMU.
The IOMMU hardware block allows physically noncontiguous pages to back up virtually
contiguous memory. Memory translation logic in the IOMMU is the same as the logic in the
CPU MMU.
The IOMMU page fault is the most commonly seen IOMMU issue. The IOMMU page fault occurs
when the page to be accessed is mapped in the page table, but is not found in memory. The
fault handler receives the context bank instance of the IOMMU and dumps out the registers
for this context.
The following log indicates an IOMMU page
fault.
[ 47.228992] msm_iommu_v1: Unexpected IOMMU page fault!
[ 47.233115] msm_iommu_v1: name = mdp_iommu
[ 47.237238] msm_iommu_v1: context = mdp_0 (0)
[ 47.241507] msm_iommu_v1: Interesting registers:
[ 47.246149] msm_iommu_v1: FAR = 0000000000000000
[ 47.250970] msm_iommu_v1: PAR = 0000000000000000
[ 47.255834] msm_iommu_v1: FSR = 00000002 [TF ]
[ 47.260540] msm_iommu_v1: FSYNR0 = 000005a1 FSYNR1 = 00030005
[ 47.266528] msm_iommu_v1: TTBR0 = 0000000071a28000
[ 47.271370] msm_iommu_v1: TTBR1 = 0000000000000000
[ 47.276248] msm_iommu_v1: SCTLR = 00001043 ACTLR = 70000000
[ 47.282221] msm_iommu_v1: CBAR = 00000000 CBFRSYNRA = 00000000
[ 47.288521] msm_iommu_v1: PRRR = ff0a81a8 NMRR = 40e040e0
[ 47.294461] msm_iommu_v1: NOTE: Value actually unknown for CBAR
[ 47.300394] msm_iommu_v1: NOTE: Value actually unknown for CBFRSYNRA
[ 47.306717] msm_iommu_v1: Page table in DDR shows PA = 0Copy to clipboard
The following table describes the fields captured in the log message.
Table : Information in IOMMU page fault log
| Item | Description |
| --- | --- |
| `name` | Name of the hardware block that caused the fault. |
| `FAR` | Fault address register (FAR) indicates the address at which the faultTranslation fault (TF) Access permission fault (APF) Stalled status (SS) -0 [001] d.h4 7524.424196: workqueue_queue_work: work struct=c10c32a8 function=def_work_fn workqueue=f55c7880 req_cpu=1 cpu=4
e.process.gapps-4758 [001] dNs4 7524.454227: workqueue_queue_work: work struct=c4727fa4 function=free_css_set_work workqueue=f6427d80 req_cpu=1 cpu=1
Binder_D-1693 [001] d.s3 7524.504198: workqueue_queue_work: work struct=cec6275c function=do_dbs_timer workqueue=f5424680 req_cpu=1 cpu=1
<...>-4832 [001] d.h3 7524.574194: workqueue_queue_work: work struct=c10c32a8 function=def_work_fn workqueue=f55c7880 req_cpu=1 cpu=4Copy to clipboard
### Enable IRQ trace
To enable interrupt tracing, run the following commands:
mount -t debugfs none /sys/kernel/debugCopy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/irq/irq_handler_entry/enableCopy to clipboard
cat /sys/kernel/debug/tracing/traceCopy to clipboard
cat /sys/kernel/debug/tracing/trace_pipeCopy to clipboard
Sample output:
TASK-PID CPU# TIMESTAMP FUNCTION
| | | | |
adbd-302 [000] 295.075568: irq_handler_entry: irq=132 name=msm_otg
adbd-302 [000] 295.075599: irq_handler_entry: irq=132 name=msm_hsusb
adbd-302 [000] 295.075782: irq_handler_entry: irq=132 name=msm_otg
adbd-302 [000] 295.075782: irq_handler_entry: irq=132 name=msm_hsusb
-0 [000] 295.076270: irq_handler_entry: irq=132 name=msm_otg
-0 [000] 295.076270: irq_handler_entry: irq=132 name=msm_hsusb
-0 [000] 295.077155: irq_handler_entry: irq=18 name=gp_timer
-0 [000] 295.087166: irq_handler_entry: irq=18 name=gp_timer
-0 [000] 295.097146: irq_handler_entry: irq=18 name=gp_timerCopy to clipboard
### Enable scheduler trace
To trace context switches between tasks, run the following commands to enable the
`sched_switch` tracer:
mount -t debugfs none /sys/kernel/debugCopy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enableCopy to clipboard
cat /sys/kernel/debug/tracing/traceCopy to clipboard
cat /sys/kernel/debug/tracing/trace_pipeCopy to clipboard
Sample
output:
TASK-PID CPU# TIMESTAMP FUNCTION
| | | | |
WindowManagerPo-484 [000] 994.583135: sched_switch: prev_comm=WindowManagerPo
SurfaceFlinger-334 [000] 994.583652: sched_switch: prev_comm=SurfaceFlinger
WindowManagerPo-484 [000] 994.584320: sched_switch: prev_comm=WindowManagerPo
GL updater-675 [000] 994.584370: sched_switch: prev_comm=GL updater
WindowManagerPo-484 [000] 994.584424: sched_switch: prev_comm=WindowManagerPoCopy to clipboard
### Find IRQ latency
To find the maximum IRQ latency and pre-emption latency in the system, enable ftrace
configurations for `IRQOFF` and `PREEMPTIONOFF` as
follows:
CONFIG_IRQSOFF_TRACER =Y
CONFIG_PREEMPT_TRACER =YCopy to clipboard
For this configuration to take effect, recompile the
kernel. This configuration detects latency in milliseconds effectively.
To enable tracing, run the following commands:
echo 0 > /sys/kernel/debug/tracing/tracing_enabledCopy to clipboard
echo 0 > /sys/kernel/debug/tracing/tracing_max_latencyCopy to clipboard
echo irqsoff > /sys/kernel/debug/tracing/current_tracerCopy to clipboard
echo 1 > /sys/kernel/debug/tracing/tracing_enabledCopy to clipboard
cat /sys/kernel/debug/tracing/traceCopy to clipboard
To find the maximum latency observed in the system, configure
`tracing_max_latency = 0`. To detect any latency higher than the
specified limit, set the threshold level in microseconds. For example,
echo 2000 > /sys/kernel/debug/tracing/tracing_threshCopy to clipboard
The following example shows the traces for IRQ latency of 16 ms:
cat /sys/kernel/debug/tracing/trace
# tracer: irqsoff
#
# WARNING: FUNCTION TRACING IS CORRUPTED
# MAY BE MISSING FUNCTION EVENTS
# irqsoff latency trace v1.1.5 on 3.4.0-perf-g7736d93-dirty
# --------------------------------------------------------------------
# latency: 16757 us, #4/4, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:1)
# -----------------
# | task: EventThread-534 (uid:1000 nice:-9 policy:0 rt_prio:0)
# -----------------
# => started at: _raw_spin_lock_irqsave
# => ended at: _raw_spin_unlock_irqrestore
#
#
# _------=> CPU#
# / _-----=> irqs-off
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# \ / ||||| \ | /
<...>-534 0d... 0us!: _raw_spin_lock_irqsave
<...>-534 0d..1 16756us+: _raw_spin_unlock_irqrestore
<...>-534 0d..1 16758us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore
<...>-534 0d..1 16780us :
=> trace_hardirqs_on
=> _raw_spin_unlock_irqrestore
=> clk_enable
=> mdss_dsi_clk_enable
=> mdss_dsi_clk_ctrl
=> mdss_dsi_clk_req
=> mdss_dsi_event_handler
=> mdss_mdp_ctl_intf_event
=> mdss_mdp_cmd_add_vsync_handler
=> mdss_mdp_overlay_vsync_ctrl
=> mdss_mdp_overlay_ioctl_handler
=> mdss_fb_ioctl
=> do_fb_ioctl
=> fb_ioctl
=> do_vfs_ioctl
=> sys_ioctl
=> ret_fast_syscall
/sys/kernel/debug/tracing #Copy to clipboard
For more information, see Documentation/trace/ftrace.txt available
at [https://www.kernel.org/doc/Documentation](https://www.kernel.org/doc/Documentation).
## Collect and parse RAM dump
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
A RAM dump is a snapshot of the system memory at the time of device failure and is useful
for debugging various crash issues.
### Enable RAM dump
By default, the RAM dump is enabled in the `debug` build, but it is
disabled in the `perf`/non-DEBUG build using the kernel command-line
parameter. To configure the RAM dump, use the
`qcom_scm.download_mode` parameter in the
meta-qcom-hwe/conf/machine/include/qcom-qcs6490.incfile
as:
- 0 – Disable
- 1 – Enable
Note: For QCS9075, configure the
`qcom_scm.download_mode` parameter in the
meta-qcom-hwe/conf/machine/include/qcom-qcs9100.inc
file.
### Collect RAM dump
To collect the RAM dump over USB, use the Product Configuration Assistant Tool
(PCAT). Download the PCAT tool from the [Qualcomm Package Manager](https://qpm.qualcomm.com/#/main/tools/find). After installing PCAT, access the PCAT user
guide at the following locations:
- Windows host: C:\Program
Files(x86)\Qualcomm\PCAT\Docs
- Linux host: /opt/qcom/PCAT/Docs/
To collect RAM dump using PCAT, do the following:
1. Launch PCAT on a Windows machine and connect the device, which is already in
the RAM dump mode.
If necessary, to trigger a device crash, click
**CRASH** on the PCAT interface.
After the device is connected to the Windows machine, PCAT
automatically starts capturing the RAM dump and dump collection progress
is displayed on the PCAT UI.
2. To view the dump, click **OPEN**.
The following is an example showing the contents of a RAM dump
directory:
Note: The Linux host uses a
command-line interface, instead of the GUI for the PCAT-based RAM dump capture,
such as `PCAT -PLUGIN CC -DEVICE -DUMPDIR /tmp -RESET TRUE
-UNIQUETS TRUE`.
## Parse RAM dump using RAMParser
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
Linux RAM dump parser (RAMParser) is an open-source tool used to parse RAM dump on
Qualcomm Linux devices. The RAMParser processes the RAM dump using the Linux kernel
symbol file, which includes vmlinux and kernel object modules, and extracts useful
information such as process stacks, IRQ, and workqueues.
Note: The RAMParser tool is built and verified only for
Windows.
### Prerequisites
- RAM dump and the corresponding vmlinux file
- Software images and scripts
- Windows machine
- Python 3.7 or a later
version
python -m pip install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org prettytableCopy to clipboard
python -m pip install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org pyelftoolsCopy to clipboard
- Pyelftools package
1. Download Pyelftools package from [https://github.com/eliben/pyelftools](https://github.com/eliben/pyelftools).
2. Unzip the downloaded file and look for
pyelftools-master>elftools directory.
3. Copy elftools directory to <installed Python
path>\Lib\site-packages.
- RAMParser software
Download the RAMParser software from [https://git.codelinaro.org/clo/la/platform/vendor/qcom-opensource/tools/-/tree/opensource-tools.lnx.1.0](https://git.codelinaro.org/clo/la/platform/vendor/qcom-opensource/tools/-/tree/opensource-tools.lnx.1.0).
Note: The RAMParser software must be always present
in the C drive on a Windows machine.
- (Optional) TRACE32 software 2023.12 or a later version on C drive
(C:\T32)
The TRACE32 is used to load the RAM dump
in the TRACE32 simulator using the RAMParser output files.
### Set up toolchains
The RAMParser requires access to gdb and nm tools. You can specify the paths to the
gdb and nm tools in one of the following ways:
- Using `--gdb-path` and `--nm-path` to specify the
absolute path
- Using `CROSS_COMPILE` to specify the prefix
- Using local\_settings.py file
Note: Availability of gdb or nm only on the path is not
supported because there are too many variations on the names to invoke.
To set up toolchains using the local\_settings.pyfile, do the
following:
1. Create a directory named ramparser\_utils\utils in the
root directory.
2. Download the toolchain in the ramparser\_utils\utils
directory from [https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads](https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads).
### local\_settings.py
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
The RAMParser attempts to determine most of the settings automatically. However, there
are some settings that are unique to the runtime environment. These unique settings are
specified in the local\_settings.py file. Because the
local\_settings.py is a Python file, it can take advantage of
the Python features.
Note: For RAMParser to pick the correct path
for toolchain utilities, add the local\_settings.py file to the
path <root>/tools/linux-ramdump-parser-v2
directory.
The format of the local\_settings.py file
is:
= Copy to clipboard
The local\_settings.py file supports the following features:
Table : Supported features in local_settings.py file
| Feature | Description |
| --- | --- |
| gdb\_path | Absolute path to the gdb tool for the RAM dump |
| nm\_path | Absolute path to the nm tool for the RAM dump |
| gdb64\_path | Absolute path to the 64‑bit gdb tool for the RAM dump |
| nm64\_path | Absolute path to the 64‑bit nm tool for the RAM dump |
| objdump\_path | Absolute path to object dump tool for the RAM dump |
| objdump64\_path | Absolute path to the 64‑bit object dump tool for the RAM --auto-dump --force-hardware --mod_path -o Copy to clipboard
- ``: This string specifies the hardware ID of the
chipset. Refer to the following table and use the appropriate value.
| Chipset | Hardware name |
| --- | --- |
| QCS6490 | `qcm6490` |
| QCS5430 | `qcm6490` |
| QCS9075 | `qcs9100` |
| | |
- python ramparse.py: This file invokes the RAMParser.
- `mod_path`: Specify this option to copy all the unstripped kernel
modules into a directory and reference it as the symbol path for the
`--mod_path` option.
- ``: Specify parser options to extract specific
data from the RAM dump. Additionally, you can pass subparser options in the
command to extract the relevant data. For example, to extract Ftrace
information, pass `--dump-ftrace` along with the related
arguments and run the
command:
python \tools\linux-ramdump-parser-v2\ramparse.py --vmlinux --auto-dump --force-hardware --dump-ftrace --ftrace-args=rwmmio --ftrace_buffer_size_kb 4096 --mod_path -o Copy to clipboard
Similarly
to extract the kconfig information, pass `--print-kconfig`. There
are multiple options available to parse the dumps and store the output in a
directory.
To view all the available commands and options, run the following
command:
python ramparse.py --helpCopy to clipboard
### RAMParser logs
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
The RAMParser generates an extensive amount of data regarding workqueues, processes
states, and call stacks. The following table lists the important files that are
generated as part of RAMParser execution.
| Filename | Description |
| --- | --- |
| dmesg\_TZ.txt | This file contains kernel logs, runqueues, workqueues, and IRQ/vmlinux /DDRCS0_0.BIN@0x80000000,/DDRCS1_0.BIN@0x100000000,/DDRCS1_1.BIN@0x180000000 -m vabits_actual=48 -m max_physmem_bits=48 -m kimage_voffset=0xffffb40bbf600000 --kaslr=0x340c3d320000Copy to clipboard
- Replace `` with the vmlinux RAM dump path.
- `kimage_voffset`, `kaslr` are extracted from the
RAMparser output file, dmesg\_TZ.txt.
- DDR offsets such as `0x80000000` are extracted from the
dump\_info.txt file available in the RAM dump collected
using PCAT.
Sample output:
crash 8.0.4
Copyright (C) 2002-2022 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
NOTE: setting vabits_actual to: 48
NOTE: setting max_physmem_bits to: 48
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=aarch64-elf-linux".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
".
For help on input options, enter "help input".
For help on output options, enter "help output".Copy to clipboard
crash> logCopy to clipboard
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
[ 0.000000] Linux version 6.6.17-debug (oe-user@oe-host) (aarch64-qcom-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Mon Mar 25 04:52:52 UTC 2024
[ 0.000000] KASLR enabled
[ 0.000000] Machine model: Qualcomm Technologies, Inc. Robotics RB3gen2 addons vision mezz platform
[ 0.000000] efi: EFI v2.7 by Qualcomm Technologies, Inc.
[ 0.000000] efi: MEMATTR=0x9ccf6018 INITRD=0x9cceaf18 RNG=0x9cce3018 MEMRESERVE=0x9ccea818
[ 0.000000] random: crng init done
[ 0.000000] Reserved memory: created CMA memory pool at 0x00000000fd000000, size 12 MiB
[ 0.000000] OF: reserved mem: initialized node adsp-heap, compatible id shared-dma-poolCopy to clipboard
crash> p memdumpCopy to clipboard
memdump = $1 = {
table_phys = 4110417920,
table = 0xffff80008087d000
}Copy to clipboard
For more information about the crash utility, see the following:
- [https://crash-utility.github.io/](https://crash-utility.github.io/)
- [https://man7.org/linux/man-pages/man8/crash.8.html](https://man7.org/linux/man-pages/man8/crash.8.html)
## Subsystem dumps
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html](https://docs.qualcomm.com/doc/80-70015-12/topic/debugging_linux_kernel.html)
This feature is available to licensed developers with authorized access. For information
on how to enable and capture core dumps of subsystems, see [Qualcomm Linux Debug Guide - Addendum](https://docs.qualcomm.com/bundle/resource/topics/80-70015-12A/debugging_linux_kernel.html#subsystem_ram_dumps).
Last Published: Oct 14, 2024
[Previous Topic
Debug Linux user space issues](https://docs.qualcomm.com/bundle/publicresource/80-70015-12/topics/using_open_source_debug_tools.md) [Next Topic
Debug common system issues](https://docs.qualcomm.com/bundle/publicresource/80-70015-12/topics/general_system_debugging.md)