# Debug Linux kernel space issues
At a high-level, you can categorize issues in the kernel space as kernel panic and bugs. To debug kernel space issues, you can use the following logs, command line tools, and dumps shown in the figure:
- The logs section includes dynamic debug, kernel debugging configuration, and function tracer.
- Command-line tools feature the /proc and debugfs file systems.
- The dumps section lists RAM dump, RAM parser, crash utility, and OpenOCD that are used for analyzing system crashes and memory states.
**Figure : Resources to debug issues in kernel space**
You can generate kernel logs using the `dmesg` command.
To debug issues in the kernel space, it's recommended to use the `debug` build. For more information about how to generate the `debug` build, see [Qualcomm Linux metadata layers overview](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-27/qualcomm_linux_metadata_layers.html).
For more information about kernel source configuration files, see [Qualcomm Linux Kernel Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-3/kernel_landing_page.html).
## Enable dynamic debug
The debugfs file system allows you to debug the kernel by enabling logs at runtime. When verifying a particular scenario, use the debugfs file system to enable logs for the specific time.
By default, the system disables the dynamic debugfs. To enable debugfs in the kernel, do the following:
1. Enable the `CONFIG_DYNAMIC_DEBUG` kernel configuration option.
2. Recompile and reflash the build.
3. To mount the debugfs file system, run the following commands:
mount -o rw,remount /
Copy to clipboard
mount -t debugfs none /sys/kernel/debug
Copy to clipboard
Verify whether the dynamic debug is allowed by ensuring the following node exists:
cd /sys/kernel/debug/dynamic_debug
Copy to clipboard
If the node exists, you must verify the defined logs:
cat /sys/kernel/debug/dynamic_debug/control
Copy to clipboard
4. Enable the debug log for the files or the function that requires debugging.
For example:
- To enable all debugfs logs in the `mdp.c` file, run the following command:
echo 'file mdp.c +p' > /sys/kernel/debug/dynamic_debug/control
Copy to clipboard
- To enable the log at line 2921 in the `mdp.c` file, run the following command:
echo 'file mdp.c line 2921 +p' > /sys/kernel/debug/dynamic_debug/control
Copy to clipboard
5. Verify logs using the `dmesg` command or run the following command:
cat /proc/kmsg
Copy to clipboard
For more information, see [Documentation/dynamic-debug-howto.txt](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/dynamic-debug-howto.rst).
## Enable kernel debugging options
You can enable the kernel configuration options to debug various issues such as memory leak, lock-related, and mutex problems. To see the available kernel debugging options, invoke `menuconfig`. For more information about kernel debugging options, see [Configure the kernel](https://docs.qualcomm.com/bundle/publicresource/topics/80-70022-3/kernel-development.html#configure-the-kernel).
The following are some debugging options:
Kernel hacking
[\*] Kernel debugging
[\*] Detect Soft Lockups
[ ] Collect scheduler statistics
[\*] Debug slab memory allocations
[\*] Memory leak debugging
[\*] Mutex debugging, deadlock detection
[\*] Spinlock debugging
[\*] Sleep-inside-spinlock checking
[ ] kobject debugging
[ ] Highmem debugging
[ ] Compile the kernel with debug info
Note
If you enable any of these debugging options, the kernel slows down marginally. Therefore, if you notice any decrease in performance, disable the kernel debug configuration options.
## Configure the kernel for debugging
The following table lists the common kernel configuration options that are useful for debugging.
| Kernel debug configuration option | Description |
| --- | --- |
| CONFIG\_DEBUG\_LIST | This option turns on checks for performing standard linked list manipulations with the `list.h` header file. If the pointers don't match, the system prints a warning, followed by a `BUG_ON` crash. |
| CONFIG\_PAGE\_POISONING | This option fills the pages with the poison pattern `(PAGE_POISON 0xaa)`, after calling `free_pages()`. |
| CONFIG\_DEBUG\_PAGEALLOC | This option verifies the patterns before calling `alloc_pages()`. |
| CONFIG\_DEBUG\_USER | This option prints a message when the system kills a user-space process due to a segmentation fault (segfault) or an invalid instruction, such as `user_debug=31` in the `arch/arm/Kconfig.debug` file.
Add the Kernel boot parameter to the `BoardConfig.mk` file. |
| CONFIG\_DEBUG\_SPINLOCK | This option identifies missing spinlock initialization and common spinlock errors, such as:
> > >
>
Waiting for more than one second on a spinlock
>
Freeing an already freed lock
>
Reinitializing a lock that was already used
>
|
| CONFIG\_DEBUG\_MUTEXES | This option detects Mutex semantic violations. |
| DEBUG\_LOCK\_ALLOC | This option detects wrong freeing of live locks. |
| CONFIG\_SLUB
CONFIG\_SLUB\_DEBUG | This option performs extra checks to detect the corruption of internal kernel memory allocation structures by adding poison for `use-after-free (0x6b)` and `buffer-overflow-padding (0xbb)`. |
| Kernel debug configuration options for extra debugging that can be verbose and can make the system slow. | Kernel debug configuration options for extra debugging that can be verbose and can make the system slow. |
| CONFIG\_DEBUG\_ATOMIC\_SLEEP | This option causes routines that may sleep to become noisy when they're called within the atomic sections. |
| DEBUG\_SPINLOCK\_SLEEP | This option causes routines that may sleep to become noisy when they're called with a spinlock held. |
| CONFIG\_DEBUG\_VM, CONFIG\_DEBUG\_HIGHMEM | This option provides an extra debugging support for virtual memory management corruption issues. |
| CONFIG\_DEBUG\_OBJECTS | This option tracks the lifetime of various objects and validates the operations on those objects. |
The following are example logs indicating common memory issues.
### List corruptions
Enabling the `CONFIG_DEBUG_LIST` option helps you identify the following crash signatures that indicate list corruption issues.
**Sample crash signature 1**
>
>
> <4> WARNING: at kernel/lib/list\_debug.c:60 \_\_list\_del\_entry+0xa0/0xd0()
>
>
> <6> list\_del corruption. prev->next should be c6fc374c, but was c18d804c
>
>
> <6> Modules linked in: adsprpc
>
>
> <6> [<c010cc94>] (unwind\_backtrace+0x0/0x138) from [<c018ad54>] (warn\_slowpath\_common+0x4c/0x64)
>
>
> <6> [<c018ad54>] (warn\_slowpath\_common+0x4c/0x64) from [<c018ae00>] (warn\_slowpath\_fmt+0x30/0x40)
>
>
> <6> [<c018ae00>] (warn\_slowpath\_fmt+0x30/0x40) from [<c03b3128>] (\_\_list\_del\_entry+0xa0/0xd0)
>
>
> <6> [<c03b3128>] (\_\_list\_del\_entry+0xa0/0xd0) from [<c01bc968>] (account\_entity\_dequeue+0x84/0x94)
>
>
> <6> [<c01bc968>] (account\_entity\_dequeue+0x84/0x94) from [<c01bdaa4>] (dequeue\_task\_fair+0x64/0x190)
**Sample crash signature 2**
>
>
> <4>WARNING: at kernel/lib/list\_debug.c:52 \_\_list\_del\_entry+0x8c/0xac()
>
>
> <4> list\_del corruption, e240c500->prev is LIST\_POISON2 (00200200)
>
>
> <4> Modules linked in: wlan(PO) cfg80211 adsp\_loader
>
>
> <4> [<c001498c>] (unwind\_backtrace+0x0/0x11c)
>
>
> from [<c0068c90>] (warn\_slowpath\_common+0x4c/0x64)
>
>
> <4> [<c0068c90>] (warn\_slowpath\_common+0x4c/0x64)
>
>
> from [<c0068d28>] (warn\_slowpath\_fmt+0x2c/0x3c)
>
>
> <4> [<c0068d28>] (warn\_slowpath\_fmt+0x2c/0x3c)
>
>
> from [<c023ef94>] (\_\_list\_del\_entry+0x8c/0xac)
>
>
> <4> [<c023ef94>] (\_\_list\_del\_entry+0x8c/0xac)
>
>
> from [<c023efc0>] (list\_del+0xc/0x24)
>
>
> <4> [<c023efc0>] (list\_del+0xc/0x24)
>
>
> from [<c043c78c>] (binder\_thread\_read+0x488/0xb70)
>
>
> <4> [<c043c78c>] (binder\_thread\_read+0x488/0xb70)
>
>
> from [<c043d094>] (binder\_ioctl+0x220/0x5b8)
**Sample crash signature 3**
>
>
> <4> WARNING: at kernel/lib/list\_debug.c:47 \_\_list\_del\_entry+0x90/0xb0()
>
>
> <4> list\_del corruption, d6ed3720->next is LIST\_POISON1 (00100100)
>
>
> <4> Modules linked in:
>
>
> <4> [] (unwind\_backtrace+0x0/0x12c) from [] (warn\_slowpath\_common+0x4c/0x64)
>
>
> <4> [] (warn\_slowpath\_common+0x4c/0x64) from [] (warn\_slowpath\_fmt+0x2c/0x3c)
>
>
> <4> [] (warn\_slowpath\_fmt+0x2c/0x3c) from [] (\_\_list\_del\_entry+0x90/0xb0)
>
>
> <4> [] (\_\_list\_del\_entry+0x90/0xb0) from [] (list\_del+0xc/0x28)
>
>
> <4> [] (list\_del+0xc/0x28) from [] (bam\_mux\_write\_done+0x34/0x11c)
### Spinlock corruption issues
Enabling the `CONFIG_DEBUG_SPINLOCK` and `CONFIG_DEBUG_MUTEXES` kernel configuration options helps you identify crash signatures that indicate spinlock corruption issues.
**Crash signature 1**
>
>
> <0>BUG: spinlock lockup on CPU#1, ndroid.launcher/1071
>
>
> <0> lock: 0xd5e8f480, .magic: dead4ead, .owner: <none>/-1, .owner\_cpu: -1
>
>
> <4> [<c0734d84>] (spin\_dump+0x74/0x84) from [<c028f8fc>]
>
>
> (do\_raw\_spin\_lock+0x144/0x188)
>
>
> <4> [<c028f8fc>] (do\_raw\_spin\_lock+0x144/0x188) from [<c0332594>] (kgsl\_mmu\_pt\_get\_flags+0x18/0x44)
>
>
> <4> [<c0332594>] (kgsl\_mmu\_pt\_get\_flags+0x18/0x44) from [<c033c6c8>] (adreno\_ringbuffer\_submitcmd+0x168/0x228)
>
>
> <4> [<c033c6c8>] (adreno\_ringbuffer\_submitcmd+0x168/0x228) from [<c033d9a4>] (sendcmd+0x3c/0x254)
**Crash signature 2**
>
>
> <0> BUG: spinlock lockup on CPU#2, kworker/2:0H/2910
>
>
> <0> lock: kpss\_clock\_reg\_lock+0x0/0x10, .magic: dead4ead, .owner:
>
>
> kworker/3:0H/2904, .owner\_cpu: 3
>
>
> <6> kworker/2:0H (2910): undefined instruction: pc=c0963098
>
>
> kernel BUG at kernel/lib/spinlock\_debug.c:95!
>
>
>
>
>
> <4> [<c0963098>] (spin\_dump+0x7c/0x94) from [<c038e030>]
>
>
> (do\_raw\_spin\_lock+0xcc/0x164)
>
>
> <4> [<c038e030>] (do\_raw\_spin\_lock+0xcc/0x164) from [<c0972228>] (\_raw\_spin\_lock\_irqsave+0x20/0x28)
>
>
> <4> [<c0972228>] (\_raw\_spin\_lock\_irqsave+0x20/0x28) from [<c011d4ac>] (\_\_kpss\_mux\_set\_sel+0x14/0x80)
>
>
> <4> [<c011d4ac>] (\_\_kpss\_mux\_set\_sel+0x14/0x80) from [<c011d54c>] (kpss\_mux\_set\_sel+0x18/0x20)
### Slub poisoning issues
Enabling the `CONFIG_SLUB` and `CONFIG_SLUB_DEBUG` kernel configuration options helps you identify crash signatures that indicate a slub poisoning issue.
<3>[ 3438.930472] =====================================================================
<3>[ 3438.937628] BUG kmalloc-64 (Tainted: G W O): Poison overwritten
<3>[ 3438.944223] ---------------------------------------------------------------------
<3>[ 3438.944228]
<3>[ 3438.953861] INFO: 0xce308408-0xce30840b. First byte 0x0 instead of 0x6b
<3>[ 3438.960470] INFO: Allocated in kgsl\_ioctl\_drawctxt\_create+0x2c/0x2ac age=4426 cpu=0 pid=21702
<3>[ 3438.968970] \_\_slab\_alloc.isra.37.constprop.43+0x4d4/0x534
<3>[ 3438.974435] kmem\_cache\_alloc\_trace+0x240/0x258
<3>[ 3438.978947] kgsl\_ioctl\_drawctxt\_create+0x2c/0x2ac
<3>[ 3438.983721] kgsl\_ioctl+0xfc/0x324
<3>[ 3438.987108] do\_vfs\_ioctl+0x80/0x54c
<3>[ 3438.990669] sys\_ioctl+0x38/0x5c
<3>[ 3438.993880] ret\_fast\_syscall+0x0/0x30
<3>[ 3438.997615] INFO: Freed in kgsl\_release+0xb8/0xc0 age=10 cpu=3 pid=21842
<3>[ 3439.004298] \_\_slab\_free+0x30/0x308
<3>[ 3439.007770] kgsl\_release+0xb8/0xc0
<3>[ 3439.011242] fput+0xcc/0x23c
<3>[ 3439.014106] filp\_close+0x68/0x80
<3>[ 3439.017407] put\_files\_struct+0xd8/0x110
<3>[ 3439.021312] do\_exit+0x164/0x860
<3>[ 3439.024524] do\_group\_exit+0x3c/0xb0
<3>[ 3439.028084] get\_signal\_to\_deliver+0x2c4/0x59c
<3>[ 3439.032510] do\_signal+0x90/0x480
<3>[ 3439.035809] do\_notify\_resume+0x50/0x5c
<3>[ 3439.039629] work\_pending+0x24/0x28
<3>[ 3439.043101] INFO: Slab 0xc0fbcd20 objects=16 used=16 fp=0x (null) flags=0x0080
<3>[ 3439.050401] INFO: Object 0xce308400 @offset=1024 fp=0xce308900
<3>[ 3439.050405]
<3>[ 3439.057694] Bytes b4 ce3083f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
<3>[ 3439.066457] Object ce308400: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b kkkkkkkk....kkkk
<3>[ 3439.075051] Object ce308410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
<3>[ 3439.083639] Object ce308420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
<3>[ 3439.092234] Object ce308430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
<3>[ 3439.100828] Redzone ce308440: bb bb bb bb ....
<3>[ 3439.108463] Padding ce3084e8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
<3>[ 3439.117145] Padding ce3084f8: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
<4>[ 3439.125144] [<c0014f78>] (unwind\_backtrace+0x0/0x138) from [<c0150250>] (check\_bytes\_and\_report+0xc0/0xe4)
<4>[ 3439.134771] [<c0150250>] (check\_bytes\_and\_report+0xc0/0xe4) from [<c015042c>] (check\_object+0x1b8/0x214)
<4>[ 3439.144237] [<c015042c>] (check\_object+0x1b8/0x214) from [<c071b0bc>] (alloc\_debug\_processing+0x7c/0x150)
<4>[ 3439.153782] [<c071b0bc>] (alloc\_debug\_processing+0x7c/0x150) from [<c071bcc0>] (\_\_slab\_alloc.isra.37.constprop.43+0x4d4/0x534)
<4>[ 3439.165153] [<c071bcc0>] (\_\_slab\_alloc.isra.37.constprop.43+0x4d4/0x534) from [<c015221c>] (kmem\_cache\_alloc\_trace+0x240/0x258)
<4>[ 3439.176611] [<c015221c>] (kmem\_cache\_alloc\_trace+0x240/0x258) from [<c014668c>] (\_\_get\_vm\_area\_node.isra.26+0x84/0x174)
<4>[ 3439.187376] [<c014668c>] (\_\_get\_vm\_area\_node.isra.26+0x84/0x174) from [<c0147038>] (get\_vm\_area\_caller+0x44/0x4c)
<4>[ 3439.197615] [<c0147038>] (get\_vm\_area\_caller+0x44/0x4c) from [<c014739c>] (vmap+0x50/0x90)
<4>[ 3439.205866] [<c014739c>] (vmap+0x50/0x90) from [<c032a694>] (\_kgsl\_sharedmem\_page\_alloc+0x238/0x3d4)
<4>[ 3439.214979] [<c032a694>] (\_kgsl\_sharedmem\_page\_alloc+0x238/0x3d4) from [<c0323e44>] (\_gpumem\_alloc+0xb0/0xfc)
<4>[ 3439.224874] [<c0323e44>] (\_gpumem\_alloc+0xb0/0xfc) from [<c0323ed0>] (kgsl\_ioctl\_gpumem\_alloc\_id+0x40/0x1a8)
<4>[ 3439.234684] [<c0323ed0>] (kgsl\_ioctl\_gpumem\_alloc\_id+0x40/0x1a8) from [<c0321874>] (kgsl\_ioctl+0xfc/0x324)
<4>[ 3439.244319] [<c0321874>] (kgsl\_ioctl+0xfc/0x324) from [<c0167fe8>] (do\_vfs\_ioctl+0x80/0x54c)
<4>[ 3439.252738] [<c0167fe8>] (do\_vfs\_ioctl+0x80/0x54c) from [<c01684ec>] (sys\_ioctl+0x38/0x5c)
<4>[ 3439.260986] [<c01684ec>] (sys\_ioctl+0x38/0x5c) from [<c000eb00>] (ret\_fast\_syscall+0x0/0x30)
<3>[ 3439.269400] FIX kmalloc-64: Restoring 0xce308408-0xce30840b=0x6b
### Page poisoning issues
Enabling the `CONFIG_DEBUG_PAGEALLOC` and `CONFIG_PAGE_POISONING` kernel configuration options helps you identify crash signature that indicate a page poisoning issue.
<1> Unable to handle kernel paging request at virtual address aaaaaaae
<1> pgd = e98b4000
<1> [aaaaaaae] [\*](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#id2)pgd=00000000
<0> Internal error: Oops: 5 [#1] PREEMPT SMP ARM
<4> Modules linked in: adsp\_loader exfat\_fs(P) exfat\_core(P)
<4> CPU: 1 Tainted: P W (3.4.0-628250-eng #1)
<4> PC is at pid\_nr\_ns+0xc/0x3c
<4> LR is at do\_task\_stat+0x248/0x83c
<4> pc : [<c00abda8>] lr : [<c018c15c>] psr: a0000093
## CPU parameters
Various CPU parameters such as core frequency, CPU governor, and cpuidle
states help you better understand the system and tune it as needed.
For more information about CPU parameters, see [kernel documentation](https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-devices-system-cpu).
## Assess memory usage
Linux uses a virtual memory system. Therefore, the addresses that the user program accesses don't correspond to the physical addresses that the hardware uses directly. Virtual memory introduces a layer of indirection, allowing programs to assign extra memory beyond the physically available memory.
Memory management implementation covers the following areas:
- Management of physical pages in the memory
- Buddy system to assign memory in large chunks
- Slab, slub, and slob allocators to assign smaller chunks of memory
- vmalloc mechanism to assign noncontiguous blocks of memory
- Address space of the processes
### /proc file system
The `/proc` file system provides the following files:
- [/proc/meminfo](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#section-n10011-n1000e-n10001)
- [/proc/vmstat](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#section-n1002b-n1000e-n10001)
- [/proc/iomem](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#section-n10045-n1000e-n10001)
- [/proc/vmallocinfo](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#section-n10053-n1000e-n10001)
### /proc/meminfo
This file provides information about distribution and usage of memory. To view the contents of this file, run the following command:
cat /proc/meminfo
Copy to clipboard
Sample output:
MemTotal: 3813532 kB
MemFree: 624836 kB
MemAvailable: 2098008 kB
Buffers: 40416 kB
Cached: 1484320 kB
SwapCached: 0 kB
Active: 1334816 kB
.
.
.
.
NFS\_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2431048 kB
Committed\_AS: 99995284 kB
VmallocTotal: 258867136 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
CmaTotal: 163840 kB
CmaFree: 1368 kB
For more information about the parameters available in this file, see [/proc filesystem](https://docs.kernel.org/filesystems/proc.html).
### /proc/vmstat
This file shows detailed virtual memory statistics from the kernel. Most of the statistics are available only if you enable the `CONFIG_VM_EVENT_COUNTERS` option in the `init/Kconfig` file.
To view the contents of this file, run the following command:
cat /proc/vmstat
Copy to clipboard
Sample output:
nr\_free\_pages 156290
nr\_alloc\_batch 132
nr\_inactive\_anon 108
nr\_active\_anon 165006
nr\_inactive\_file 212275
nr\_active\_file 168709
nr\_unevictable 64
nr\_mlock 64
nr\_anon\_pages 164982
nr\_mapped 90366
nr\_file\_pages 381184:
:
unevictable\_pgs\_mlocked 0
unevictable\_pgs\_munlocked 0
unevictable\_pgs\_cleared 0
unevictable\_pgs\_stranded 0
unevictable\_pgs\_mlockfreed 0
For more information about the parameters available in this file, see [Linux manual page](https://man7.org/linux/man-pages/man8/vmstat.8.html).
### /proc/iomem
This file shows the memory map of the system for its various device drivers. To view the contents of this file, run the following command:
cat /proc/iomem
Copy to clipboard
Sample output:
007781b8-007791b7 : vmpm
010aa000-010abfff : tsens\_physical
010ac000-010ac003 : pshold-base
010ad000-010aefff : tsens\_physical
01680000-0168ffff : /soc/arm,smmu-[anoc1@1680000](mailto:anoc1%401680000)
016c0000-016fffff : /soc/arm,smmu-[anoc2@16c0000](mailto:anoc2%4016c0000)
01d0101c-01d0101f : sp2soc\_irq\_status
01d01024-01d01027 : sp2soc\_irq\_clr
01d01028-01d0102b : sp2soc\_irq\_mask
.
.
.
0caa0000-0caa3fff : jpeg\_hw
0caa4000-0caa47ff : fd\_core
0caa5000-0caa53ff : fd\_misc
0cd00000-0cd3ffff : /soc/arm,smmu-[mmss@cd00000](mailto:mmss%40cd00000)
17817000-17817fff : msm-watchdog
17900000-1790dfff : msm-gladiator-erp
80000000-857fffff : System RAM
80080000-817fffff : Kernel code
82330000-82945fff : Kernel data
88f00000-8aafffff : System RAM
95300000-17e3bffff : System RAM
### /proc/vmallocinfo
This file shows detailed information about virtual address allocation through `vmalloc` or `ioremap`. To view the contents of this file, run the following command:
cat /proc/vmallocinfo
Copy to clipboard
Sample output:
0xbf000000-0xbf002000 8192 module\_alloc\_update\_bounds+0xc/0x5c pages=1 vmalloc
0xbf004000-0xbf008000 16384 module\_alloc\_update\_bounds+0xc/0x5c pages=3 vmalloc
0xee800000-0xef800000 16777216 iotable\_init+0x0/0xb0 phys=36800000 ioremap
0xf0000000-0xf0002000 8192 of\_iomap+0x30/0x38 ioremap
0xf0002000-0xf0004000 8192 of\_iomap+0x30/0x38 ioremap
0xf0004000-0xf000c000 32768 gen\_pool\_add\_virt+0x48/0xb8 pages= 7 vmalloc
0xf000c000-0xf000e000 8192 msm\_pm\_setup\_saved\_state+0xcc/0x1bc ioremap
………………………
0xf0174000-0xf0176000 8192 msm\_cpu\_status\_probe+0xd8/0x20c ioremap
0xf0f24000-0xf0f28000 16384 \_kgsl\_sharedmem\_page\_alloc+0xa0/0x41c pages=3 vmalloc
0xf0f39000-0xf0f3e000 20480 \_kgsl\_sharedmem\_page\_alloc+0xa0/0x41c pages=4 vmalloc
0xf0f61000-0xf0f66000 20480 \_kgsl\_sharedmem\_page\_alloc+0xa0/0x41c pages=4 vmalloc
……….
0xfa400000-0xfa600000 2097152 iotable\_init+0x0/0xb0 phys=fa00000 ioremap
0xfa71e000-0xfa71f000 4096 iotable\_init+0x0/0xb0 phys=f991e000 ioremap
0xfefd8000-0xff000000 163840 pcpu\_get\_vm\_areas+0x0/0x56c vmalloc
### memblock interface
The `memblock` interface on the debugfs file system provides details about the available and reserved memory regions in the system. This interface provides the following files:
- [/sys/kernel/debug/memblock/memory](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#section-n10015-n1000e-n10001)
- [/sys/kernel/debug/memblock/reserved](https://docs.qualcomm.com/doc/80-70022-12/topic/debugging_linux_kernel.html#section-n10038-n1000e-n10001)
### /sys/kernel/debug/memblock/memory
This file provides information about all the available memory regions (HLOS and non-HLOS) visible to the Linux kernel. To determine the overall memory accessible to the kernel, calculate the difference between the start and end addresses of each of the region and sum these values to get the total occupied RAM. The remaining memory, which is the difference between the RAM size of the device and the occupied RAM, is the non-HLOS memory or the memory occupied by other subsystems.
To view the contents of this file, run the following command:
cat /sys/kernel/debug/memblock/memory
Copy to clipboard
Sample output:
0: 0x0000000080000000..0x00000000857fffff
1: 0x0000000088f00000..0x000000008aafffff
2: 0x0000000095300000..0x000000017e3bffff
### /sys/kernel/debug/memblock/reserved
This file provides information about all the reserved memory regions in the system.
To view the contents of this file, run the following command:
cat /sys/kernel/debug/memblock/reserved
Copy to clipboard
Sample output:
0: 0x0000000080080000..0x0000000082944fff
1: 0x0000000083200000..0x0000000083259bb4
2: 0x0000000083400000..0x00000000839506c9
3: 0x00000000f5800000..0x00000000ffbfffff
4: 0x00000000ffff7000..0x00000000ffffefff
5: 0x00000000ffffff40..0x00000000ffffff77
6: 0x00000000ffffff80..0x00000000ffffffb7
7: 0x00000000ffffffc0..0x00000000fffffff7
8: 0x0000000179258000..0x000000017d9fffff
9: 0x000000017da17000..0x000000017da1ffff
10: 0x000000017da20e00..0x000000017da26fff
11: 0x000000017da27300..0x000000017da2735f
12: 0x000000017da27380..0x000000017da273df
13: 0x000000017da27400..0x000000017da2755f
14: 0x000000017da27580..0x000000017da27587
15: 0x000000017da275c0..0x000000017da275c7
16: 0x000000017da29600..0x000000017da29924
17: 0x000000017da29940..0x000000017da29c64
18: 0x000000017da29c80..0x000000017da29fa4
19: 0x000000017da29fac..0x000000017da2a3f8
20: 0x000000017da2a3fc..0x000000017da2a42e
21: 0x000000017da2a430..0x000000017da2a45e
22: 0x000000017da2a460..0x000000017e3bffff
### Memory leak
Enable the following configuration options to debug kernel memory leak issues:
- `CONFIG_DEBUG_KMEMLEAK=y`
- `CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE= 4000`
- `CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y`
By default, a kernel thread scans the memory every 10 minutes and prints the number of new unreferenced objects found. For example,
unreferenced object 0xec26f000 (size 4096):
comm "Binder\_2", pid 4592, jiffies 8848 (age 336.710s)
hex dump (first 32 bytes):
ec 4d f8 c0 02 00 00 00 00 00 00 00 00 00 00 00 .M..............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<c0126f70>] kmem\_cache\_alloc\_trace+0x17c/0x238
[<c03059c4>] ddl\_client\_transact+0xd0/0x158
[<c0314a3c>] ddl\_open+0x4c/0x194
[<c0302288>] vcd\_init\_client\_context+0x14/0x9c
[<c02ffd10>] vcd\_open\_in\_ready+0x3c/0x94
[<c02fd31c>] vcd\_open+0x214/0x274
[<c031b244>] vid\_dec\_open\_client+0x1d0/0x288
[<c031b3ac>] vid\_dec\_open+0x30/0x7c
[<c012ff70>] chrdev\_open+0x10c/0x134
[<c012aa7c>] \_\_dentry\_open.isra.12+0x190/0x29c
[<c0138a78>] do\_last.isra.29+0x690/0x6c0
[<c0138c70>] path\_openat+0xb8/0x35c
[<c0138ff4>] do\_filp\_open+0x2c/0x78
[<c012b7a0>] do\_sys\_open+0xd8/0x170
[<c000df20>] ret\_fast\_syscall+0x0/0x30
[<ffffffff>] 0xffffffff
Disable the `KMEMLEAK` option at boot time by passing `KMEMLEAK=off` on the kernel command line.
For more information about the `kmemleak.txt` file, see [kernel documentation](https://www.kernel.org/doc/Documentation).
The following extra kernel configuration options are available to track the allocator of each page of memory:
- `CONFIG_PAGE_OWNER`
- `CONFIG_PAGE_OWNER_ENABLE_DEFAULT`
- `CONFIG_PAGE_EXTENSION`
Enabling these options to parse all pages can help identify several allocations, which may indicate a memory leak issue.
### Identify memory corruption issues
Enable the following kernel configuration options to identify memory corruption issues:
- `CONFIG_PAGE_POISONING`
- `CONFIG_SLUB_DEBUG_ON`
- `CONFIG_DEBUG_LIST`
- `CONFIG_SLUB_DEBUG`
Sample log:
BUG <slab cache affected>: <What went wrong>
-----------------------------------------------
INFO: <corruption start>-<corruption\_end> <more info>
INFO: Slab <address> <slab information>
INFO: Object <address> <object information>
INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by cpu> pid=<pid of the process>
INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu> pid=<pid of the process>
For more information about slub debugging, see `Documentation/vm/slub.txt` available at [kernel documentation](https://www.kernel.org/doc/Documentation).
### Out of memory
When the system fails to assign a page, the kernel logs display a message such as the following:
<4>[12146.861355] Thread-430: page allocation failure: order:0, mode:0x10d2
<CALL STACK>
<4>[12146.951687] Mem-info:
<4>[12146.953909] Normal per-cpu:
<4>[12146.956686] CPU 0: hi: 186, btch: 31 usd: 61
<4>[12146.961489] CPU 1: hi: 186, btch: 31 usd: 0
<4>[12146.966235] HighMem per-cpu:
<4>[12146.969122] CPU 0: hi: 186, btch: 31 usd: 54
<4>[12146.973877] CPU 1: hi: 186, btch: 31 usd: 0
…….
<4>[12147.010770] Normal free:53192kB min:3508kB low:4384kB high:5260kB ……..
<4>[12147.050805] lowmem\_reserve[]: 0 9022 9022
<4>[12147.054610] HighMem free:153360kB min:512kB low:1824kB high:3140kB ………
<4>[12147.095453] lowmem\_reserve[]: 0 0 0
<4>[12147.098617] Normal: 118\*4kB 232\*8kB 161\*16kB 110\*32kB 34\*64kB 9\*128kB 10\*256kB 6\*512kB 7\*1024kB 6\*2048kB 4\*4096kB = 53224kB
<4>[12147.115455] HighMem: 2774\*4kB 11769\*8kB 3005\*16kB 1\*32kB 0\*64kB 0\*128kB 0\*256kB 0\*512kB 0\*1024kB 0\*2048kB 0\*4096kB = 153360kB
………………
These messages indicate that the system couldn't assign the requested page. The top line of the log message provides several details about the out-of-memory issue. For example, in the following log:
<4>[1214.855361] Thread-4: page allocation failure: order:2, mode:0x10d2
- **order**: Indicates the size of the page. In this example, 2 ^2^ x PAGE\_SIZE (4 K) = 16 K
- Linux uses a buddy allocator that allocates pages in powers of 2.
- The maximum size of the buddy allocator is the order of 10 = 2 ^10^ x 4 kB = 4 MB.
- For allocation > 4 MB, use an alternate allocation method such as the contiguous memory allocator (CMA).
- For failure of higher-order allocations, examine whether the memory can be virtually contiguous, instead of being physically contiguous.
- **mode**: Indicates the type of page requested
- `mode` provides information about get free pages (GFP) flags. In this example, `mode` is `0x10d2`.
- `mode` is the result of OR operation on all the GFP flags in the allocation.
- Pages available in the system
A page allocation failure message prints details about the size of pages that were available in the system.
Normal: 118\*4kB 232\*8kB 161\*16kB 110\*32kB 34\*64kB 9\*128kB 10\*256kB 6\*512kB 7\*1024kB 6\*2048kB 4\*4096kB = 53224kB
HighMem: 2774\*4kB 11769\*8kB 3005\*16kB 1\*32kB 0\*64kB 0\*128kB 0\*256kB 0\*512kB 0\*1024kB 0\*2048kB 0\*4096kB = 153360kB
### IOMMU page fault
IOMMU, also known as the system MMU (SMMU), performs memory management functions on behalf of subsystems that don't have their own MMU.
The IOMMU hardware block allows physically noncontiguous pages to support virtually contiguous memory. The memory translation logic in the IOMMU is the same as the logic in the CPU MMU.
The IOMMU page fault is the most common IOMMU issue. The fault occurs when the requested page is mapped in the page table but isn't found in the memory. The fault handler receives the context bank instance of the IOMMU and dumps the registers for this context.
The following log indicates an IOMMU page fault.
[ 47.228992] msm\_iommu\_v1: Unexpected IOMMU page fault!
[ 47.233115] msm\_iommu\_v1: name = mdp\_iommu
[ 47.237238] msm\_iommu\_v1: context = mdp\_0 (0)
[ 47.241507] msm\_iommu\_v1: Interesting registers:
[ 47.246149] msm\_iommu\_v1: FAR = 0000000000000000
[ 47.250970] msm\_iommu\_v1: PAR = 0000000000000000
[ 47.255834] msm\_iommu\_v1: FSR = 00000002 [TF ]
[ 47.260540] msm\_iommu\_v1: FSYNR0 = 000005a1 FSYNR1 = 00030005
[ 47.266528] msm\_iommu\_v1: TTBR0 = 0000000071a28000
[ 47.271370] msm\_iommu\_v1: TTBR1 = 0000000000000000
[ 47.276248] msm\_iommu\_v1: SCTLR = 00001043 ACTLR = 70000000
[ 47.282221] msm\_iommu\_v1: CBAR = 00000000 CBFRSYNRA = 00000000
[ 47.288521] msm\_iommu\_v1: PRRR = ff0a81a8 NMRR = 40e040e0
[ 47.294461] msm\_iommu\_v1: NOTE: Value actually unknown for CBAR
[ 47.300394] msm\_iommu\_v1: NOTE: Value actually unknown for CBFRSYNRA
[ 47.306717] msm\_iommu\_v1: Page table in DDR shows PA = 0
The following table describes the fields captured in the log message.
Table : Information in IOMMU page fault log
| Item | Description |
| --- | --- |
| `name` | Name of the hardware block that caused the fault. |
| `FAR` | Fault address register (FAR) indicates the address at which the fault occurred. |
| `FSR` | Fault status register (FSR) indicates the following:
> > >
>
Translation fault (TF)
>
Access permission fault (APF)
>
Stalled status (SS)
>
|
The FSR is one of the most important registers in IOMMU debugging. This register has read/write-clear access. The read operation on this register reads the value in the register while the write operation clears the bits corresponding to 1s in the written data and leaves the bits corresponding to 0s unchanged. This process prevents inadvertent clearing of new faults when writing the register to clear an old fault. Some useful bits in this register are:
Table : Bits in fault status register
| Bit | Description |
| --- | --- |
| `[Bit 1]: TF` | Translation fault (invalid page table entry) |
| `[Bit 2]: AFF` | Access fault |
| `[Bit 3]: APF` | Permission fault (write to read only region) |
| `[Bit 4]: TLBMF` | TLB miss fault |
| `[Bit 5]: HTWDEEF` | Hardware table walk decode error external fault |
| `[Bit 6]: HTWSEEF` | Hardware table walk subordinate error external fault |
| `[Bit 7]: MHF` | Many hits in TLB |
| `[Bit 16]: SL` | Second-level fault (fault occurred in second level of page table) |
| `[Bit 30]: SS` | Stalled status |
| `[Bit 31]: MULTI` | Multiple faults |
The `TF`, `APF`, and `SL` flags indicate normal operation, whereas `TLBMF`, `HTWDEEF`, `HTWSEEF`, and `MHF` flags indicate that there is an issue.
### IOMMU page table
The IOMMU page table dump provides a faulting address from the FAR and the register dump in the kernel log. This faulting address represents the virtual address, and you can acquire the corresponding physical address from the page table. From the page table dump, you can identify whether the requested address is mapped or not mapped. Each IOMMU domain has a page table, and the dump includes page tables for each of the domains. Currently, there are six domains.
The following is the sample dump of the `Domain: 2` page table.
Domain: 2 [L2 cache redirect for page tables is OFF]
0x00000000--0x0001ffff [0x00020000] [UNMAPPED]
0x00020000--0x01807fff [0x017e8000] A:0x82a8e000--0x84275fff [0x017e8000] [R/W][4K]
0x01808000--0x01939fff [0x00132000] A:0xf0c24000--0xf0d55fff [0x00132000] [R/W][4K]
0x0193a000--0x0199ffff [0x00066000] A:0xf13fa000--0xf145ffff [0x00066000] [R/W][4K]
0x019a0000--0x01e85fff [0x004e6000] A:0xf966e000--0xf9b53fff [0x004e6000] [R/W][4K]
0x01e86000--0x01ffffff [0x0017a000] [UNMAPPED]
0x02000000--0x02feffff [0x00ff0000] A:0xf5a22000--0xf6a11fff [0x00ff0000] [R/W][4K]
In this example, the first column represents the virtual address, the second column represents the number of bytes in the corresponding region of contiguous physical addresses, and the third column represents the physical addresses. The permissions are also mentioned for each of these regions.
### Memory map
For more information about the memory map, see the latest Release Notes.
### Stack corruption
Wrong coding logic accesses memory locations in the stack, leading to changes in values at those memory locations, causing stack corruption. Stack corruption can occur in the following ways:
- Due to bad code logic, the program consumes all the stack memory, and it writes memory beyond the stack boundaries, causing a stack overflow.
- Accessing an array that's out of bounds.
- An undefined or freed pointer that points at a stack address.
- Corrupted return address of a caller function.
To identify the stack corruption issues, enable the following kernel configuration options:
- `CONFIG_STACKPROTECTOR`
- `CONFIG_STACKPROTECTOR_STRONG`
The Kernel address sanitizer (KASAN) utility also helps in identifying
some stack corruption issues.
## Trace kernel functions
Function tracer (ftrace) provides tracing utilities to perform system-wide profiling and tracing at runtime.
To use ftrace, enable the following configuration options:
- `CONFIG_FTRACE`
- `CONFIG_HAVE_FUNCTION_TRACER`
- `CONFIG_HAVE_FUNCTION_GRAPH_TRACER`
- `CONFIG_HAVE_DYNAMIC_FTRACE`
- `CONFIG_HAVE_FTRACE_MCOUNT_RECORD`
The following are some operations that ftrace can perform to debug kernel issues:
### Dump ftrace information to kmsg buffer
To dump the ftrace information into the kmsg buffer anytime from the source code, call the `ftrace_dump(DUMP_ALL)` function.
To increase the buffer size of the ftrace ring, run the following command:
echo 200 > /sys/kernel/debug/tracing/buffer_size_kb
Copy to clipboard
### Enable work queue trace
To enable work queue tracing, run the following commands:
mount -t debugfs none /sys/kernel/debug
Copy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/workqueue/enable
Copy to clipboard
echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
Copy to clipboard
cat /sys/kernel/debug/tracing/trace_pipe
Copy to clipboard
cat /sys/kernel/debug/tracing/per_cpu/cpu1/trace
Copy to clipboard
Sample output:
# tracer: nop
#
# entries-in-buffer/entries-written: 8682/8682 #P:1
#
# \_-----=> irqs-off
# / \_----=> need-resched
# | / \_---=> hardirq/softirq
# || / \_--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<...>-4783 [001] d.s4 7524.354249: workqueue\_queue\_work: work struct=f2d91ee4 function=free\_css\_set\_work workqueue=f6427d80 req\_cpu=1 cpu=1
<idle>-0 [001] d.h4 7524.424196: workqueue\_queue\_work: work struct=c10c32a8 function=def\_work\_fn workqueue=f55c7880 req\_cpu=1 cpu=4
e.process.gapps-4758 [001] dNs4 7524.454227: workqueue\_queue\_work: work struct=c4727fa4 function=free\_css\_set\_work workqueue=f6427d80 req\_cpu=1 cpu=1
Binder\_D-1693 [001] d.s3 7524.504198: workqueue\_queue\_work: work struct=cec6275c function=do\_dbs\_timer workqueue=f5424680 req\_cpu=1 cpu=1
<...>-4832 [001] d.h3 7524.574194: workqueue\_queue\_work: work struct=c10c32a8 function=def\_work\_fn workqueue=f55c7880 req\_cpu=1 cpu=4
### Enable Interrupt request (IRQ) trace
To enable interrupt tracing, run the following commands:
mount -t debugfs none /sys/kernel/debug
Copy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/irq/irq_handler_entry/enable
Copy to clipboard
cat /sys/kernel/debug/tracing/trace
Copy to clipboard
cat /sys/kernel/debug/tracing/trace_pipe
Copy to clipboard
Sample output:
TASK-PID CPU# TIMESTAMP FUNCTION
| | | | |
adbd-302 [000] 295.075568: irq\_handler\_entry: irq=132 name=msm\_otg
adbd-302 [000] 295.075599: irq\_handler\_entry: irq=132 name=msm\_hsusb
adbd-302 [000] 295.075782: irq\_handler\_entry: irq=132 name=msm\_otg
adbd-302 [000] 295.075782: irq\_handler\_entry: irq=132 name=msm\_hsusb
<idle>-0 [000] 295.076270: irq\_handler\_entry: irq=132 name=msm\_otg
<idle>-0 [000] 295.076270: irq\_handler\_entry: irq=132 name=msm\_hsusb
<idle>-0 [000] 295.077155: irq\_handler\_entry: irq=18 name=gp\_timer
<idle>-0 [000] 295.087166: irq\_handler\_entry: irq=18 name=gp\_timer
<idle>-0 [000] 295.097146: irq\_handler\_entry: irq=18 name=gp\_timer
### Enable scheduler trace
To trace context switches between tasks, run the following commands to
enable the `sched_switch` tracer:
mount -t debugfs none /sys/kernel/debug
Copy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
Copy to clipboard
cat /sys/kernel/debug/tracing/trace
Copy to clipboard
cat /sys/kernel/debug/tracing/trace_pipe
Copy to clipboard
Sample output:
TASK-PID CPU# TIMESTAMP FUNCTION
| | | | |
WindowManagerPo-484 [000] 994.583135: sched\_switch: prev\_comm=WindowManagerPo
SurfaceFlinger-334 [000] 994.583652: sched\_switch: prev\_comm=SurfaceFlinger
WindowManagerPo-484 [000] 994.584320: sched\_switch: prev\_comm=WindowManagerPo
GL updater-675 [000] 994.584370: sched\_switch: prev\_comm=GL updater
WindowManagerPo-484 [000] 994.584424: sched\_switch: prev\_comm=WindowManagerPo
### Find IRQ latency
To find the maximum IRQ latency and pre-emption latency in the system, enable ftrace configurations for `IRQOFF` and `PREEMPTIONOFF` as follows:
CONFIG_IRQSOFF_TRACER =Y
CONFIG_PREEMPT_TRACER =Y
Copy to clipboard
For this configuration to take effect, recompile the kernel. This configuration detects latency in milliseconds effectively.
To enable tracing, run the following commands:
echo 0 > /sys/kernel/debug/tracing/tracing_enabled
Copy to clipboard
echo 0 > /sys/kernel/debug/tracing/tracing_max_latency
Copy to clipboard
echo irqsoff > /sys/kernel/debug/tracing/current_tracer
Copy to clipboard
echo 1 > /sys/kernel/debug/tracing/tracing_enabled
Copy to clipboard
cat /sys/kernel/debug/tracing/trace
Copy to clipboard
To find the maximum latency observed in the system, configure `tracing_max_latency = 0`. To detect any latency higher than the specified limit, set the threshold level in microseconds. For example,
echo 2000 > /sys/kernel/debug/tracing/tracing_thresh
Copy to clipboard
The following example shows the traces for IRQ latency of 16 ms:
cat /sys/kernel/debug/tracing/trace
# tracer: irqsoff
#
# WARNING: FUNCTION TRACING IS CORRUPTED
# MAY BE MISSING FUNCTION EVENTS
# irqsoff latency trace v1.1.5 on 3.4.0-perf-g7736d93-dirty
# --------------------------------------------------------------------
# latency: 16757 us, #4/4, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:1)
# -----------------
# | task: EventThread-534 (uid:1000 nice:-9 policy:0 rt\_prio:0)
# -----------------
# => started at: \_raw\_spin\_lock\_irqsave
# => ended at: \_raw\_spin\_unlock\_irqrestore
#
#
# \_------=> CPU#
# / \_-----=> irqs-off
# | / \_----=> need-resched
# || / \_---=> hardirq/softirq
# ||| / \_--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# / ||||| | /
<...>-534 0d... 0us!: \_raw\_spin\_lock\_irqsave
<...>-534 0d..1 16756us+: \_raw\_spin\_unlock\_irqrestore
<...>-534 0d..1 16758us+: trace\_hardirqs\_on <-\_raw\_spin\_unlock\_irqrestore
<...>-534 0d..1 16780us : <stack trace>
=> trace\_hardirqs\_on
=> \_raw\_spin\_unlock\_irqrestore
=> clk\_enable
=> mdss\_dsi\_clk\_enable
=> mdss\_dsi\_clk\_ctrl
=> mdss\_dsi\_clk\_req
=> mdss\_dsi\_event\_handler
=> mdss\_mdp\_ctl\_intf\_event
=> mdss\_mdp\_cmd\_add\_vsync\_handler
=> mdss\_mdp\_overlay\_vsync\_ctrl
=> mdss\_mdp\_overlay\_ioctl\_handler
=> mdss\_fb\_ioctl
=> do\_fb\_ioctl
=> fb\_ioctl
=> do\_vfs\_ioctl
=> sys\_ioctl
=> ret\_fast\_syscall
/sys/kernel/debug/tracing #
For more information, see the `Documentation/trace/ftrace.txt` file available at [kernel documentation](https://www.kernel.org/doc/Documentation).
## Collect and parse RAM dump
A RAM dump is a snapshot of the system memory at the time of device failure and is useful for debugging various crash issues.
### Enable RAM dump
By default, the `debug` build incorporates the RAM dump. However, it's disabled in the `perf/non-DEBUG` build using the kernel command-line parameter. To configure the RAM dump, use the `qcom_scm.download_mode` parameter in the `meta-qcom-hwe/conf/machine/include/` file as:
- 0: Disable
- 1: Enable
The following table lists the `.inc` file corresponding to different chipsets:
| Chipset | Filename |
| --- | --- |
| QCS5430/QCS6490 | `meta-qcom-hwe/conf/machine/include/qcom-qcs6490.inc` |
| IQ-8275 | `meta-qcom-hwe/conf/machine/include/qcom-qcs8300.inc` |
| IQ-9075 | `meta-qcom-hwe/conf/machine/include/qcom-qcs9100.inc` |
| IQ-615 | `meta-qcom-hwe/conf/machine/include/qcom-qcs615.inc` |
### Collect RAM dump
To collect the RAM dump over USB, use the Product Configuration Assistant Tool (PCAT). Download the PCAT tool from the [Qualcomm Software Center](https://softwarecenter.qualcomm.com/#/catalog/item/PCAT). After installing PCAT, access the PCAT user guide at the following locations:
- Windows host: `C:\Program Files(x86)\Qualcomm\PCAT\Docs`
- Linux host: `/opt/qcom/PCAT/Docs/`
To collect RAM dump using PCAT, do the following:
1. Start PCAT on a Windows host and connect the device, which is already in the RAM dump mode.
If necessary, to trigger a device crash, select **CRASH** on the PCAT interface.
After you connect the device to the Windows host, PCAT automatically starts capturing the RAM dump, and the PCAT UI displays the progress of the dump collection.
2. To view the dump, select **OPEN**.
The following is an example showing the contents of a RAM dump directory:
Note
The Linux host uses a command-line interface, instead of the GUI for the PCAT-based RAM dump capture, such as
`PCAT -PLUGIN CC -DEVICE -DUMPDIR /tmp -RESET TRUE -UNIQUETS TRUE`.
## Parse RAM dumps using RAMParser
Linux RAM dump parser (RAMParser) is an open-source tool used to parse RAM dump on Qualcomm Linux devices. The RAMParser processes the RAM dump using the Linux kernel symbol file, which includes vmlinux and kernel object modules, and extracts useful information such as process stacks, IRQ, and work queues.
Note
The RAMParser tool is built and verified for Windows only.
### Prerequisites
- RAM dump and the corresponding `vmlinux` file
- Software images and scripts
- Windows PC
- Python 3.7 or a later version
python -m pip install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org prettytable
Copy to clipboard
python -m pip install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org pyelftools
Copy to clipboard
- Pyelftools package
1. Download Pyelftools package from [https://github.com/eliben/pyelftools](https://github.com/eliben/pyelftools).
2. Unzip the downloaded file and look for `pyelftools-master>elftools` directory.
3. Copy elftools directory to `\Lib\site-packages`.
- RAMParser software
Download the RAMParser software from [https://git.codelinaro.org/clo/la/platform/vendor/qcom-opensource/tools/-/tree/opensource-tools.lnx.1.0](https://git.codelinaro.org/clo/la/platform/vendor/qcom-opensource/tools/-/tree/opensource-tools.lnx.1.0).
Note
The RAMParser software must always be present in the C drive on a Windows host.
- (Optional) TRACE32 software 2023.12 or a later version on C drive (`C:\T32`)
The TRACE32 software loads the RAM dump in the TRACE32 simulator using the RAMParser output files.
### Set up toolchains
The RAMParser requires access to gdb and nm tools. You can specify the paths to the gdb and nm tools in one of the following ways:
- Using `--gdb-path` and `--nm-path` to specify the absolute path
- Using `CROSS_COMPILE` to specify the prefix
- Using `local_settings.py` file
Note
Availability of gdb or nm only on the path isn't supported because there are too many variations on the names to invoke.
To set up toolchains using the `local_settings.py` file, do the following:
1. Create a directory named `ramparser_utils\utils` in the root directory.
2. Download the toolchain (aarch64-none-linux-gnu) in the `ramparser_utils\utils` directory from [https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads](https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads).
### local\_settings.py
The RAMParser automatically determines most of the settings. However, some settings are unique to the runtime environment. You specify these unique settings in the
`local_settings.py` file. As `local_settings.py` is a Python file, it can leverage Python features.
Note
For RAMParser to pick the correct path for toolchain utilities, add the `local_settings.py` file to the path `/tools/linux-ramdump-parser-v2` directory.
The format of the `local_settings.py` file is:
=
Copy to clipboard
The `local_settings.py` file supports the following features:
Table : Supported features in local_settings.py file
| Feature | Description |
| --- | --- |
| gdb\_path | Absolute path to the gdb tool for the RAM dump |
| nm\_path | Absolute path to the nm tool for the RAM dump |
| gdb64\_path | Absolute path to the 64‑bit gdb tool for the RAM dump |
| nm64\_path | Absolute path to the 64‑bit nm tool for the RAM dump |
| objdump\_path | Absolute path to the object dump tool for the RAM dump |
| objdump64\_path | Absolute path to the 64‑bit object dump tool for the RAM dump |
### Example: local\_settings.py
import parser_util,os,sys
path = os.path.abspath(os.path.dirname(__file__))
ramparser_utils_path = os.path.abspath(os.path.join(path, "../../ramparser_utils"))
nm_path = ramparser_utils_path + "\\utils\\arm-none-eabi-nm.exe"
gdb_path = ramparser_utils_path + "\\utils\\arm-none-eabi-gdb.exe"
objdump_path = ramparser_utils_path + "\\utils\\objdump.exe"
nm64_path = ramparser_utils_path + "\\utils\\aarch64-linux-gnu-nm.exe"
gdb64_path = ramparser_utils_path + "\\utils\\mingw64\\bin\\gdb-multiarch.exe"
objdump64_path = ramparser_utils_path + "\\utils\\all-objdump.exe"
Copy to clipboard
### RAMParser commands
To parse dumps using RAMParser, run the following command in the Windows Shell:
python ramparse.py --vmlinux --auto-dump --force-hardware --mod_path -o