# Debug Linux kernel space issues
At a high-level, issues in the kernel space are categorized as kernel panic and bugs. The following figure shows the logs, command-line options, and dumps that can be used to debug issues in the kernel space.
**Figure : Resources to debug issues in kernel space**
You can generate kernel logs using the `dmesg` command.
To debug issues in the kernel space, it is recommended to use the `debug` build. For more information on how to generate the `debug` build, see [Overview of metadata layers](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-27/platform_software_features.html#overview_of_meta_data_layers).
For information on kernel source configuration files, see [Qualcomm Linux Kernel Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-3/overview.html).
## Enable dynamic debug
The debugfs file system allows you to debug the kernel by enabling logs at runtime. When verifying a particular scenario, use the debugfs file system to enable logs for the specific time period.
By default, the dynamic debugfs is disabled. To enable debugfs in the kernel, do the following:
1. Enable the `CONFIG_DYNAMIC_DEBUG` kernel configuration option.
2. Recompile and reflash the build.
3. To mount the debugfs file system, run the following commands:
mount -o rw,remount /
Copy to clipboard
mount -t debugfs none /sys/kernel/debug
Copy to clipboard
To verify if the dynamic debug is enabled or not, verify that the following node exists:
cd /sys/kernel/debug/dynamic_debug
Copy to clipboard
If the node exists, verify logs that are defined:
cat /sys/kernel/debug/dynamic_debug/control
Copy to clipboard
4. Enable the debug log for the files or the function that requires debugging.
For example:
- To enable all debugfs logs in the `mdp.c` file, run the following command:
echo 'file mdp.c +p' > /sys/kernel/debug/dynamic_debug/control
Copy to clipboard
- To enable the log at line 2921 in the `mdp.c` file, run the following command:
echo 'file mdp.c line 2921 +p' > /sys/kernel/debug/dynamic_debug/control
Copy to clipboard
5. Verify logs using the `dmesg` command or run the following command:
cat /proc/kmsg
Copy to clipboard
For more information, see [Documentation/dynamic-debug-howto.txt](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/dynamic-debug-howto.rst).
## Enable kernel debugging options
The kernel configuration options can be enabled to debug various issues such as memory leak, lock-related, and Mutex problems. To see the available kernel debugging options, invoke `menuconfig`. For more information about the kernel debugging options, see [Kernel configurations](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-3/customize.html#kernel-configurations).
The following are some debugging options:
Kernel hacking
[\*] Kernel debugging
[\*] Detect Soft Lockups
[ ] Collect scheduler statistics
[\*] Debug slab memory allocations
[\*] Memory leak debugging
[\*] Mutex debugging, deadlock detection
[\*] Spinlock debugging
[\*] Sleep-inside-spinlock checking
[ ] kobject debugging
[ ] Highmem debugging
[ ] Compile the kernel with debug info
Note
If you enable any of these debugging options, the kernel slows down marginally. Therefore, if you notice any decrease in performance, disable the kernel debug configuration options.
## Kernel configuration options
The following table lists the common kernel configuration options that are useful for debugging.
| Kernel debug configuration option | Description |
| --- | --- |
| CONFIG\_DEBUG\_LIST | This option turns on checks for performing standard linked list manipulations with `list.h` header file. If the pointers do not match, a warning is printed, followed by `BUG_ON` crash. |
| CONFIG\_PAGE\_POISONING | This option fills the pages with poison pattern `(PAGE_POISON 0xaa)`, after calling `free_pages()`. |
| CONFIG\_DEBUG\_PAGEALLOC | This option verifies the patterns before calling `alloc_pages()`. |
| CONFIG\_DEBUG\_USER | This option prints a message when a user space process is killed due to a segmentation fault (segfault) or an invalid instruction such as `user_debug=31` in the `arch/arm/Kconfig.debug` file.
Add the Kernel boot parameter to the `BoardConfig.mk` file. |
| CONFIG\_DEBUG\_SPINLOCK | This option identifies missing spinlock initialization and common spinlock errors, such as:
> > >
>
Waiting for more than one second on a spinlock
>
Freeing an already freed lock
>
Reinitializing a lock that was already used
>
|
| CONFIG\_DEBUG\_MUTEXES | This option detects Mutex semantic violations. |
| DEBUG\_LOCK\_ALLOC | This option detects incorrect freeing of live locks. |
| CONFIG\_SLUB
CONFIG\_SLUB\_DEBUG | This option performs additional checks to detect the corruption of internal kernel memory allocation structures by adding poison for `use-after-free (0x6b)` and `buffer-overflow-padding (0xbb)`. |
| Kernel debug configuration options for additional debugging that can be verbose and can make the system slow. | Kernel debug configuration options for additional debugging that can be verbose and can make the system slow. |
| CONFIG\_DEBUG\_ATOMIC\_SLEEP | This option causes routines that might sleep to become noisy when they are called inside atomic sections. |
| DEBUG\_SPINLOCK\_SLEEP | This option causes routines that might sleep to become noisy when they are called with a spinlock held. |
| CONFIG\_DEBUG\_VM, CONFIG\_DEBUG\_HIGHMEM | This option provides additional debugging support for virtual memory management corruptions. |
| CONFIG\_DEBUG\_OBJECTS | This option tracks the lifetime of various objects and validates the operations on those objects. |
The following are example logs indicating common memory issues.
### List corruptions
When the `CONFIG_DEBUG_LIST` option is enabled, the following crash signatures indicating a list corruption issue can be identified.
**Sample crash signature 1**
<4> WARNING: at kernel/lib/list\_debug.c:60 \_\_list\_del\_entry+0xa0/0xd0()
<6> list\_del corruption. prev->next should be c6fc374c, but was c18d804c
<6> Modules linked in: adsprpc
<6> [<c010cc94>] (unwind\_backtrace+0x0/0x138) from [<c018ad54>] (warn\_slowpath\_common+0x4c/0x64)
<6> [<c018ad54>] (warn\_slowpath\_common+0x4c/0x64) from [<c018ae00>] (warn\_slowpath\_fmt+0x30/0x40)
<6> [<c018ae00>] (warn\_slowpath\_fmt+0x30/0x40) from [<c03b3128>] (\_\_list\_del\_entry+0xa0/0xd0)
<6> [<c03b3128>] (\_\_list\_del\_entry+0xa0/0xd0) from [<c01bc968>] (account\_entity\_dequeue+0x84/0x94)
<6> [<c01bc968>] (account\_entity\_dequeue+0x84/0x94) from [<c01bdaa4>] (dequeue\_task\_fair+0x64/0x190)
**Sample crash signature 2**
<4>WARNING: at kernel/lib/list\_debug.c:52 \_\_list\_del\_entry+0x8c/0xac()
<4> list\_del corruption, e240c500->prev is LIST\_POISON2 (00200200)
<4> Modules linked in: wlan(PO) cfg80211 adsp\_loader
<4> [<c001498c>] (unwind\_backtrace+0x0/0x11c)
from [<c0068c90>] (warn\_slowpath\_common+0x4c/0x64)
<4> [<c0068c90>] (warn\_slowpath\_common+0x4c/0x64)
from [<c0068d28>] (warn\_slowpath\_fmt+0x2c/0x3c)
<4> [<c0068d28>] (warn\_slowpath\_fmt+0x2c/0x3c)
from [<c023ef94>] (\_\_list\_del\_entry+0x8c/0xac)
<4> [<c023ef94>] (\_\_list\_del\_entry+0x8c/0xac)
from [<c023efc0>] (list\_del+0xc/0x24)
<4> [<c023efc0>] (list\_del+0xc/0x24)
from [<c043c78c>] (binder\_thread\_read+0x488/0xb70)
<4> [<c043c78c>] (binder\_thread\_read+0x488/0xb70)
from [<c043d094>] (binder\_ioctl+0x220/0x5b8)
**Sample crash signature 3**
<4> WARNING: at kernel/lib/list\_debug.c:47 \_\_list\_del\_entry+0x90/0xb0()
<4> list\_del corruption, d6ed3720->next is LIST\_POISON1 (00100100)
<4> Modules linked in:
<4> [] (unwind\_backtrace+0x0/0x12c) from [] (warn\_slowpath\_common+0x4c/0x64)
<4> [] (warn\_slowpath\_common+0x4c/0x64) from [] (warn\_slowpath\_fmt+0x2c/0x3c)
<4> [] (warn\_slowpath\_fmt+0x2c/0x3c) from [] (\_\_list\_del\_entry+0x90/0xb0)
<4> [] (\_\_list\_del\_entry+0x90/0xb0) from [] (list\_del+0xc/0x28)
<4> [] (list\_del+0xc/0x28) from [] (bam\_mux\_write\_done+0x34/0x11c)
### Spinlock corruption issues
When `CONFIG_DEBUG_SPINLOCK` and `CONFIG_DEBUG_MUTEXES` kernel configuration options are enabled, the following crash signatures indicating spinlock corruption issues can be identified.
**Crash signature 1**
<0>BUG: spinlock lockup on CPU#1, ndroid.launcher/1071
<0> lock: 0xd5e8f480, .magic: dead4ead, .owner: <none>/-1, .owner\_cpu: -1
<4> [<c0734d84>] (spin\_dump+0x74/0x84) from [<c028f8fc>]
(do\_raw\_spin\_lock+0x144/0x188)
<4> [<c028f8fc>] (do\_raw\_spin\_lock+0x144/0x188) from [<c0332594>] (kgsl\_mmu\_pt\_get\_flags+0x18/0x44)
<4> [<c0332594>] (kgsl\_mmu\_pt\_get\_flags+0x18/0x44) from [<c033c6c8>] (adreno\_ringbuffer\_submitcmd+0x168/0x228)
<4> [<c033c6c8>] (adreno\_ringbuffer\_submitcmd+0x168/0x228) from [<c033d9a4>] (sendcmd+0x3c/0x254)
**Crash signature 2**
<0> BUG: spinlock lockup on CPU#2, kworker/2:0H/2910
<0> lock: kpss\_clock\_reg\_lock+0x0/0x10, .magic: dead4ead, .owner:
kworker/3:0H/2904, .owner\_cpu: 3
<6> kworker/2:0H (2910): undefined instruction: pc=c0963098
kernel BUG at kernel/lib/spinlock\_debug.c:95!
<4> [<c0963098>] (spin\_dump+0x7c/0x94) from [<c038e030>]
(do\_raw\_spin\_lock+0xcc/0x164)
<4> [<c038e030>] (do\_raw\_spin\_lock+0xcc/0x164) from [<c0972228>] (\_raw\_spin\_lock\_irqsave+0x20/0x28)
<4> [<c0972228>] (\_raw\_spin\_lock\_irqsave+0x20/0x28) from [<c011d4ac>] (\_\_kpss\_mux\_set\_sel+0x14/0x80)
<4> [<c011d4ac>] (\_\_kpss\_mux\_set\_sel+0x14/0x80) from [<c011d54c>] (kpss\_mux\_set\_sel+0x18/0x20)
### Slub poisoning issues
When `CONFIG_SLUB` and `CONFIG_SLUB_DEBUG` kernel configuration options are enabled, the following crash signature indicating a slub poisoning issue can be identified.
<3>[ 3438.930472] =====================================================================
<3>[ 3438.937628] BUG kmalloc-64 (Tainted: G W O): Poison overwritten
<3>[ 3438.944223] ---------------------------------------------------------------------
<3>[ 3438.944228]
<3>[ 3438.953861] INFO: 0xce308408-0xce30840b. First byte 0x0 instead of 0x6b
<3>[ 3438.960470] INFO: Allocated in kgsl\_ioctl\_drawctxt\_create+0x2c/0x2ac age=4426 cpu=0 pid=21702
<3>[ 3438.968970] \_\_slab\_alloc.isra.37.constprop.43+0x4d4/0x534
<3>[ 3438.974435] kmem\_cache\_alloc\_trace+0x240/0x258
<3>[ 3438.978947] kgsl\_ioctl\_drawctxt\_create+0x2c/0x2ac
<3>[ 3438.983721] kgsl\_ioctl+0xfc/0x324
<3>[ 3438.987108] do\_vfs\_ioctl+0x80/0x54c
<3>[ 3438.990669] sys\_ioctl+0x38/0x5c
<3>[ 3438.993880] ret\_fast\_syscall+0x0/0x30
<3>[ 3438.997615] INFO: Freed in kgsl\_release+0xb8/0xc0 age=10 cpu=3 pid=21842
<3>[ 3439.004298] \_\_slab\_free+0x30/0x308
<3>[ 3439.007770] kgsl\_release+0xb8/0xc0
<3>[ 3439.011242] fput+0xcc/0x23c
<3>[ 3439.014106] filp\_close+0x68/0x80
<3>[ 3439.017407] put\_files\_struct+0xd8/0x110
<3>[ 3439.021312] do\_exit+0x164/0x860
<3>[ 3439.024524] do\_group\_exit+0x3c/0xb0
<3>[ 3439.028084] get\_signal\_to\_deliver+0x2c4/0x59c
<3>[ 3439.032510] do\_signal+0x90/0x480
<3>[ 3439.035809] do\_notify\_resume+0x50/0x5c
<3>[ 3439.039629] work\_pending+0x24/0x28
<3>[ 3439.043101] INFO: Slab 0xc0fbcd20 objects=16 used=16 fp=0x (null) flags=0x0080
<3>[ 3439.050401] INFO: Object 0xce308400 @offset=1024 fp=0xce308900
<3>[ 3439.050405]
<3>[ 3439.057694] Bytes b4 ce3083f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
<3>[ 3439.066457] Object ce308400: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b kkkkkkkk....kkkk
<3>[ 3439.075051] Object ce308410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
<3>[ 3439.083639] Object ce308420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
<3>[ 3439.092234] Object ce308430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
<3>[ 3439.100828] Redzone ce308440: bb bb bb bb ....
<3>[ 3439.108463] Padding ce3084e8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
<3>[ 3439.117145] Padding ce3084f8: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
<4>[ 3439.125144] [<c0014f78>] (unwind\_backtrace+0x0/0x138) from [<c0150250>] (check\_bytes\_and\_report+0xc0/0xe4)
<4>[ 3439.134771] [<c0150250>] (check\_bytes\_and\_report+0xc0/0xe4) from [<c015042c>] (check\_object+0x1b8/0x214)
<4>[ 3439.144237] [<c015042c>] (check\_object+0x1b8/0x214) from [<c071b0bc>] (alloc\_debug\_processing+0x7c/0x150)
<4>[ 3439.153782] [<c071b0bc>] (alloc\_debug\_processing+0x7c/0x150) from [<c071bcc0>] (\_\_slab\_alloc.isra.37.constprop.43+0x4d4/0x534)
<4>[ 3439.165153] [<c071bcc0>] (\_\_slab\_alloc.isra.37.constprop.43+0x4d4/0x534) from [<c015221c>] (kmem\_cache\_alloc\_trace+0x240/0x258)
<4>[ 3439.176611] [<c015221c>] (kmem\_cache\_alloc\_trace+0x240/0x258) from [<c014668c>] (\_\_get\_vm\_area\_node.isra.26+0x84/0x174)
<4>[ 3439.187376] [<c014668c>] (\_\_get\_vm\_area\_node.isra.26+0x84/0x174) from [<c0147038>] (get\_vm\_area\_caller+0x44/0x4c)
<4>[ 3439.197615] [<c0147038>] (get\_vm\_area\_caller+0x44/0x4c) from [<c014739c>] (vmap+0x50/0x90)
<4>[ 3439.205866] [<c014739c>] (vmap+0x50/0x90) from [<c032a694>] (\_kgsl\_sharedmem\_page\_alloc+0x238/0x3d4)
<4>[ 3439.214979] [<c032a694>] (\_kgsl\_sharedmem\_page\_alloc+0x238/0x3d4) from [<c0323e44>] (\_gpumem\_alloc+0xb0/0xfc)
<4>[ 3439.224874] [<c0323e44>] (\_gpumem\_alloc+0xb0/0xfc) from [<c0323ed0>] (kgsl\_ioctl\_gpumem\_alloc\_id+0x40/0x1a8)
<4>[ 3439.234684] [<c0323ed0>] (kgsl\_ioctl\_gpumem\_alloc\_id+0x40/0x1a8) from [<c0321874>] (kgsl\_ioctl+0xfc/0x324)
<4>[ 3439.244319] [<c0321874>] (kgsl\_ioctl+0xfc/0x324) from [<c0167fe8>] (do\_vfs\_ioctl+0x80/0x54c)
<4>[ 3439.252738] [<c0167fe8>] (do\_vfs\_ioctl+0x80/0x54c) from [<c01684ec>] (sys\_ioctl+0x38/0x5c)
<4>[ 3439.260986] [<c01684ec>] (sys\_ioctl+0x38/0x5c) from [<c000eb00>] (ret\_fast\_syscall+0x0/0x30)
<3>[ 3439.269400] FIX kmalloc-64: Restoring 0xce308408-0xce30840b=0x6b
### Page poisoning issues
When `CONFIG_DEBUG_PAGEALLOC` and `CONFIG_PAGE_POISONING` kernel configuration options are enabled, the following crash signature indicating a page poisoning issue can be identified.
<1> Unable to handle kernel paging request at virtual address aaaaaaae
<1> pgd = e98b4000
<1> [aaaaaaae] [\*](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#id1)pgd=00000000
<0> Internal error: Oops: 5 [#1] PREEMPT SMP ARM
<4> Modules linked in: adsp\_loader exfat\_fs(P) exfat\_core(P)
<4> CPU: 1 Tainted: P W (3.4.0-628250-eng #1)
<4> PC is at pid\_nr\_ns+0xc/0x3c
<4> LR is at do\_task\_stat+0x248/0x83c
<4> pc : [<c00abda8>] lr : [<c018c15c>] psr: a0000093
## CPU parameters
Various CPU parameters such as core frequency, CPU governor, and cpuidle
states help you to know more about the system and tune it accordingly.
For information about CPU parameters, see [https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-devices-system-cpu](https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-devices-system-cpu).
## Memory
Linux uses a virtual memory system. Therefore, the addresses that the user program accesses do not correspond to the physical addresses that the hardware uses directly. Virtual memory introduces a layer of indirection that allows programs running on the system to allocate additional memory, apart from the physically available memory.
Memory management implementation covers the following areas:
- Management of physical pages in the memory
- Buddy system to allocate memory in large chunks
- Slab, slub, and slob allocators to allocate smaller chunks of memory
- vmalloc mechanism to allocate noncontiguous blocks of memory
- Address space of the processes
### /proc file system
The following files are available on the `/proc` file system:
- [/proc/meminfo](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#section-n10011-n1000e-n10001)
- [/proc/vmstat](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#section-n1002b-n1000e-n10001)
- [/proc/iomem](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#section-n10045-n1000e-n10001)
- [/proc/vmallocinfo](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#section-n10053-n1000e-n10001)
### /proc/meminfo
This file provides information about the distribution and usage of memory. To view the contents of this file, run the following command:
cat /proc/meminfo
Copy to clipboard
Sample output:
MemTotal: 3813532 kB
MemFree: 624836 kB
MemAvailable: 2098008 kB
Buffers: 40416 kB
Cached: 1484320 kB
SwapCached: 0 kB
Active: 1334816 kB
.
.
.
.
NFS\_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2431048 kB
Committed\_AS: 99995284 kB
VmallocTotal: 258867136 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
CmaTotal: 163840 kB
CmaFree: 1368 kB
For details about the parameters available in this file, see [https://docs.kernel.org/filesystems/proc.html](https://docs.kernel.org/filesystems/proc.html).
### /proc/vmstat
This file shows detailed virtual memory statistics from the kernel. Most of the statistics are available only if the `CONFIG_VM_EVENT_COUNTERS` option in the `init/Kconfig` file is enabled.
To view the contents of this file, run the following command:
cat /proc/vmstat
Copy to clipboard
Sample output:
nr\_free\_pages 156290
nr\_alloc\_batch 132
nr\_inactive\_anon 108
nr\_active\_anon 165006
nr\_inactive\_file 212275
nr\_active\_file 168709
nr\_unevictable 64
nr\_mlock 64
nr\_anon\_pages 164982
nr\_mapped 90366
nr\_file\_pages 381184:
:
unevictable\_pgs\_mlocked 0
unevictable\_pgs\_munlocked 0
unevictable\_pgs\_cleared 0
unevictable\_pgs\_stranded 0
unevictable\_pgs\_mlockfreed 0
For details about the parameters available in this file,
see [https://man7.org/linux/man-pages/man8/vmstat.8.html](https://man7.org/linux/man-pages/man8/vmstat.8.html).
### /proc/iomem
This file shows the memory map of the system for its various device drivers. To view the contents of this file, run the following command:
cat /proc/iomem
Copy to clipboard
Sample output:
007781b8-007791b7 : vmpm
010aa000-010abfff : tsens\_physical
010ac000-010ac003 : pshold-base
010ad000-010aefff : tsens\_physical
01680000-0168ffff : /soc/arm,smmu-[anoc1@1680000](mailto:anoc1%401680000)
016c0000-016fffff : /soc/arm,smmu-[anoc2@16c0000](mailto:anoc2%4016c0000)
01d0101c-01d0101f : sp2soc\_irq\_status
01d01024-01d01027 : sp2soc\_irq\_clr
01d01028-01d0102b : sp2soc\_irq\_mask
.
.
.
0caa0000-0caa3fff : jpeg\_hw
0caa4000-0caa47ff : fd\_core
0caa5000-0caa53ff : fd\_misc
0cd00000-0cd3ffff : /soc/arm,smmu-[mmss@cd00000](mailto:mmss%40cd00000)
17817000-17817fff : msm-watchdog
17900000-1790dfff : msm-gladiator-erp
80000000-857fffff : System RAM
80080000-817fffff : Kernel code
82330000-82945fff : Kernel data
88f00000-8aafffff : System RAM
95300000-17e3bffff : System RAM
### /proc/vmallocinfo
This file shows detailed information about virtual address allocation through `vmalloc` or `ioremap`. To view the contents of this file, run the following command:
cat /proc/vmallocinfo
Copy to clipboard
Sample output:
0xbf000000-0xbf002000 8192 module\_alloc\_update\_bounds+0xc/0x5c pages=1 vmalloc
0xbf004000-0xbf008000 16384 module\_alloc\_update\_bounds+0xc/0x5c pages=3 vmalloc
0xee800000-0xef800000 16777216 iotable\_init+0x0/0xb0 phys=36800000 ioremap
0xf0000000-0xf0002000 8192 of\_iomap+0x30/0x38 ioremap
0xf0002000-0xf0004000 8192 of\_iomap+0x30/0x38 ioremap
0xf0004000-0xf000c000 32768 gen\_pool\_add\_virt+0x48/0xb8 pages= 7 vmalloc
0xf000c000-0xf000e000 8192 msm\_pm\_setup\_saved\_state+0xcc/0x1bc ioremap
………………………
0xf0174000-0xf0176000 8192 msm\_cpu\_status\_probe+0xd8/0x20c ioremap
0xf0f24000-0xf0f28000 16384 \_kgsl\_sharedmem\_page\_alloc+0xa0/0x41c pages=3 vmalloc
0xf0f39000-0xf0f3e000 20480 \_kgsl\_sharedmem\_page\_alloc+0xa0/0x41c pages=4 vmalloc
0xf0f61000-0xf0f66000 20480 \_kgsl\_sharedmem\_page\_alloc+0xa0/0x41c pages=4 vmalloc
……….
0xfa400000-0xfa600000 2097152 iotable\_init+0x0/0xb0 phys=fa00000 ioremap
0xfa71e000-0xfa71f000 4096 iotable\_init+0x0/0xb0 phys=f991e000 ioremap
0xfefd8000-0xff000000 163840 pcpu\_get\_vm\_areas+0x0/0x56c vmalloc
### memblock interface
The `memblock` interface on the debugfs file system provides details about the available and reserved memory regions in the system. The following files are available in this interface:
- [/sys/kernel/debug/memblock/memory](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#section-n10015-n1000e-n10001)
- [/sys/kernel/debug/memblock/reserved](https://docs.qualcomm.com/doc/80-70017-12/topic/debugging_linux_kernel.html#section-n10038-n1000e-n10001)
### /sys/kernel/debug/memblock/memory
This file provides details about all the available memory regions (HLOS and non-HLOS) visible to the Linux kernel. To know the overall memory accessible to the kernel, calculate the difference between start and end addresses of each of the regions and add the values to get the total occupied RAM. The remaining memory, which is the difference between the RAM size of the device and the occupied RAM, is the non-HLOS memory or the memory occupied by other subsystems.
To view the contents of this file, run the following command:
cat /sys/kernel/debug/memblock/memory
Copy to clipboard
Sample output:
0: 0x0000000080000000..0x00000000857fffff
1: 0x0000000088f00000..0x000000008aafffff
2: 0x0000000095300000..0x000000017e3bffff
### /sys/kernel/debug/memblock/reserved
This file provides details about all reserved memory regions in the system.
To view the contents of this file, run the following command:
cat /sys/kernel/debug/memblock/reserved
Copy to clipboard
Sample output:
0: 0x0000000080080000..0x0000000082944fff
1: 0x0000000083200000..0x0000000083259bb4
2: 0x0000000083400000..0x00000000839506c9
3: 0x00000000f5800000..0x00000000ffbfffff
4: 0x00000000ffff7000..0x00000000ffffefff
5: 0x00000000ffffff40..0x00000000ffffff77
6: 0x00000000ffffff80..0x00000000ffffffb7
7: 0x00000000ffffffc0..0x00000000fffffff7
8: 0x0000000179258000..0x000000017d9fffff
9: 0x000000017da17000..0x000000017da1ffff
10: 0x000000017da20e00..0x000000017da26fff
11: 0x000000017da27300..0x000000017da2735f
12: 0x000000017da27380..0x000000017da273df
13: 0x000000017da27400..0x000000017da2755f
14: 0x000000017da27580..0x000000017da27587
15: 0x000000017da275c0..0x000000017da275c7
16: 0x000000017da29600..0x000000017da29924
17: 0x000000017da29940..0x000000017da29c64
18: 0x000000017da29c80..0x000000017da29fa4
19: 0x000000017da29fac..0x000000017da2a3f8
20: 0x000000017da2a3fc..0x000000017da2a42e
21: 0x000000017da2a430..0x000000017da2a45e
22: 0x000000017da2a460..0x000000017e3bffff
### Memory leak
To debug the kernel memory leak issues, enable the following configuration options:
- `CONFIG_DEBUG_KMEMLEAK=y`
- `CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE= 4000`
- `CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y`
By default, a kernel thread scans the memory every 10 minutes and prints the number of new unreferenced objects found. For example,
unreferenced object 0xec26f000 (size 4096):
comm "Binder\_2", pid 4592, jiffies 8848 (age 336.710s)
hex dump (first 32 bytes):
ec 4d f8 c0 02 00 00 00 00 00 00 00 00 00 00 00 .M..............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<c0126f70>] kmem\_cache\_alloc\_trace+0x17c/0x238
[<c03059c4>] ddl\_client\_transact+0xd0/0x158
[<c0314a3c>] ddl\_open+0x4c/0x194
[<c0302288>] vcd\_init\_client\_context+0x14/0x9c
[<c02ffd10>] vcd\_open\_in\_ready+0x3c/0x94
[<c02fd31c>] vcd\_open+0x214/0x274
[<c031b244>] vid\_dec\_open\_client+0x1d0/0x288
[<c031b3ac>] vid\_dec\_open+0x30/0x7c
[<c012ff70>] chrdev\_open+0x10c/0x134
[<c012aa7c>] \_\_dentry\_open.isra.12+0x190/0x29c
[<c0138a78>] do\_last.isra.29+0x690/0x6c0
[<c0138c70>] path\_openat+0xb8/0x35c
[<c0138ff4>] do\_filp\_open+0x2c/0x78
[<c012b7a0>] do\_sys\_open+0xd8/0x170
[<c000df20>] ret\_fast\_syscall+0x0/0x30
[<ffffffff>] 0xffffffff
The `KMEMLEAK` option can be disabled at boot time by passing `KMEMLEAK=off` on the kernel command line.
For details about the `kmemleak.txt` file, see [https://www.kernel.org/doc/Documentation](https://www.kernel.org/doc/Documentation).
The following are the additional kernel configuration options available to track the allocator of each page of memory:
- `CONFIG_PAGE_OWNER`
- `CONFIG_PAGE_OWNER_ENABLE_DEFAULT`
- `CONFIG_PAGE_EXTENSION`
Parsing all pages with these options enabled can help when there are multiple allocations, which might be a memory leak issue.
### Memory corruption
To identify memory corruption issues, enable the following kernel configuration options:
- `CONFIG_PAGE_POISONING`
- `CONFIG_SLUB_DEBUG_ON`
- `CONFIG_DEBUG_LIST`
- `CONFIG_SLUB_DEBUG`
Sample log:
BUG <slab cache affected>: <What went wrong>
-----------------------------------------------
INFO: <corruption start>-<corruption\_end> <more info>
INFO: Slab <address> <slab information>
INFO: Object <address> <object information>
INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by cpu> pid=<pid of the process>
INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu> pid=<pid of the process>
For more information on slub debugging, see `Documentation/vm/slub.txt` available at [https://www.kernel.org/doc/Documentation](https://www.kernel.org/doc/Documentation).
### Out of memory
When the system cannot allocate a page, the kernel logs display a message such as the following:
<4>[12146.861355] Thread-430: page allocation failure: order:0, mode:0x10d2
<CALL STACK>
<4>[12146.951687] Mem-info:
<4>[12146.953909] Normal per-cpu:
<4>[12146.956686] CPU 0: hi: 186, btch: 31 usd: 61
<4>[12146.961489] CPU 1: hi: 186, btch: 31 usd: 0
<4>[12146.966235] HighMem per-cpu:
<4>[12146.969122] CPU 0: hi: 186, btch: 31 usd: 54
<4>[12146.973877] CPU 1: hi: 186, btch: 31 usd: 0
…….
<4>[12147.010770] Normal free:53192kB min:3508kB low:4384kB high:5260kB ……..
<4>[12147.050805] lowmem\_reserve[]: 0 9022 9022
<4>[12147.054610] HighMem free:153360kB min:512kB low:1824kB high:3140kB ………
<4>[12147.095453] lowmem\_reserve[]: 0 0 0
<4>[12147.098617] Normal: 118\*4kB 232\*8kB 161\*16kB 110\*32kB 34\*64kB 9\*128kB 10\*256kB 6\*512kB 7\*1024kB 6\*2048kB 4\*4096kB = 53224kB
<4>[12147.115455] HighMem: 2774\*4kB 11769\*8kB 3005\*16kB 1\*32kB 0\*64kB 0\*128kB 0\*256kB 0\*512kB 0\*1024kB 0\*2048kB 0\*4096kB = 153360kB
………………
These messages indicate that the system could not allocate the requested page. The top line of the log message provides several details about the out-of-memory issue. For example, in the following log:
<4>[1214.855361] Thread-4: page allocation failure: order:2, mode:0x10d2
- **order**: Indicates the size of the page; in this example, 2^2 x PAGE\_SIZE (4 K) = 16 K
- Linux uses a buddy allocator that allocates pages in powers of 2.
- The maximum size of the buddy allocator is the order of 10 = 2^10 x 4 kB = 4 MB.
- For allocation > 4 MB, use an alternate allocation method such as the contiguous memory allocator (CMA).
- For failure of higher-order allocations, examine whether the memory can be virtually contiguous, instead of being physically contiguous.
- **mode**: Indicates the type of page that was requested
- `mode` provides information about get free pages (GFP) flags. In this example, `mode` is `0x10d2`.
- `mode` is the result of OR operation on all the GFP flags in the allocation.
- Pages available in the system
A page allocation failure message prints details about the size of pages that were available in the system.
Normal: 118\*4kB 232\*8kB 161\*16kB 110\*32kB 34\*64kB 9\*128kB 10\*256kB 6\*512kB 7\*1024kB 6\*2048kB 4\*4096kB = 53224kB
HighMem: 2774\*4kB 11769\*8kB 3005\*16kB 1\*32kB 0\*64kB 0\*128kB 0\*256kB 0\*512kB 0\*1024kB 0\*2048kB 0\*4096kB = 153360kB
### IOMMU page fault
IOMMU is also known as the system MMU (SMMU). It performs memory management functions on behalf of subsystems that do not have their own MMU.
The IOMMU hardware block allows physically noncontiguous pages to back up virtually contiguous memory. Memory translation logic in the IOMMU is the same as the logic in the CPU MMU.
The IOMMU page fault is the most commonly seen IOMMU issue. The IOMMU page fault occurs when the page to be accessed is mapped in the page table, but is not found in memory. The fault handler receives the context bank instance of the IOMMU and dumps out the registers for this context.
The following log indicates an IOMMU page fault.
[ 47.228992] msm\_iommu\_v1: Unexpected IOMMU page fault!
[ 47.233115] msm\_iommu\_v1: name = mdp\_iommu
[ 47.237238] msm\_iommu\_v1: context = mdp\_0 (0)
[ 47.241507] msm\_iommu\_v1: Interesting registers:
[ 47.246149] msm\_iommu\_v1: FAR = 0000000000000000
[ 47.250970] msm\_iommu\_v1: PAR = 0000000000000000
[ 47.255834] msm\_iommu\_v1: FSR = 00000002 [TF ]
[ 47.260540] msm\_iommu\_v1: FSYNR0 = 000005a1 FSYNR1 = 00030005
[ 47.266528] msm\_iommu\_v1: TTBR0 = 0000000071a28000
[ 47.271370] msm\_iommu\_v1: TTBR1 = 0000000000000000
[ 47.276248] msm\_iommu\_v1: SCTLR = 00001043 ACTLR = 70000000
[ 47.282221] msm\_iommu\_v1: CBAR = 00000000 CBFRSYNRA = 00000000
[ 47.288521] msm\_iommu\_v1: PRRR = ff0a81a8 NMRR = 40e040e0
[ 47.294461] msm\_iommu\_v1: NOTE: Value actually unknown for CBAR
[ 47.300394] msm\_iommu\_v1: NOTE: Value actually unknown for CBFRSYNRA
[ 47.306717] msm\_iommu\_v1: Page table in DDR shows PA = 0
The following table describes the fields captured in the log message.
Table : Information in IOMMU page fault log
| Item | Description |
| --- | --- |
| `name` | Name of the hardware block that caused the fault. |
| `FAR` | Fault address register (FAR) indicates the address at which the fault occurred. |
| `FSR` | Fault status register (FSR) indicates the following:
> > >
>
Translation fault (TF)
>
Access permission fault (APF)
>
Stalled status (SS)
>
|
The FSR is one of the most important registers in IOMMU debugging. This register has read/write-clear access. The read operation on this register reads the value in the register while the write operation clears the bits corresponding to 1s in the written data, and leaves the bits corresponding to 0s unchanged. This process prevents inadvertent clearing of new faults when writing the register to clear an old fault. Some of the useful bits in this register are:
Table : Bits in fault status register
| Bit | Description |
| --- | --- |
| `[Bit 1]: TF` | Translation fault (invalid page table entry) |
| `[Bit 2]: AFF` | Access fault |
| `[Bit 3]: APF` | Permission fault (write to read only region) |
| `[Bit 4]: TLBMF` | TLB miss fault |
| `[Bit 5]: HTWDEEF` | Hardware table walk decode error external fault |
| `[Bit 6]: HTWSEEF` | Hardware table walk slave error external fault |
| `[Bit 7]: MHF` | Multiple hits in TLB |
| `[Bit 16]: SL` | Second-level fault (fault occurred in second level of page table) |
| `[Bit 30]: SS` | Stalled status |
| `[Bit 31]: MULTI` | Multiple faults |
The `TF`, `APF`, and `SL` flags indicate normal operation, whereas `TLBMF`, `HTWDEEF`, `HTWSEEF`, and `MHF` flags indicate that there is an issue.
### IOMMU page table
The IOMMU page table dump provides a faulting address from the FAR and the register dump in the kernel log. This faulting address is the virtual address, and the corresponding physical address can be acquired from the page table. From the page table dump, the address being accessed can be identified as mapped or not mapped. Each IOMMU domain has a page table. The dump includes page tables for each of the domains. There are currently six domains.
The following is the sample dump of the `Domain: 2` page table.
Domain: 2 [L2 cache redirect for page tables is OFF]
0x00000000--0x0001ffff [0x00020000] [UNMAPPED]
0x00020000--0x01807fff [0x017e8000] A:0x82a8e000--0x84275fff [0x017e8000] [R/W][4K]
0x01808000--0x01939fff [0x00132000] A:0xf0c24000--0xf0d55fff [0x00132000] [R/W][4K]
0x0193a000--0x0199ffff [0x00066000] A:0xf13fa000--0xf145ffff [0x00066000] [R/W][4K]
0x019a0000--0x01e85fff [0x004e6000] A:0xf966e000--0xf9b53fff [0x004e6000] [R/W][4K]
0x01e86000--0x01ffffff [0x0017a000] [UNMAPPED]
0x02000000--0x02feffff [0x00ff0000] A:0xf5a22000--0xf6a11fff [0x00ff0000] [R/W][4K]
In this example, the first column represents the virtual address, the second column represents the number of bytes in the corresponding region of contiguous physical addresses, and the third column represents the physical addresses. The permissions are also mentioned for each of these regions.
### Memory map
For information on the memory map, see the latest Release Notes.
### Stack corruption
Stack corruption occurs when memory locations in the stack are accessed due to incorrect coding logic, which leads to a change in values at those memory locations. Stack corruption can occur in the following ways:
- Due to bad code logic, all the stack memory is consumed, and memory is written beyond the stack boundaries causing stack overflow.
- Accessing an array that is out of bounds.
- An undefined or freed pointer that points at a stack address.
- Corrupted return address of a caller function.
To identify the stack corruption issues, enable the following kernel configuration options:
- `CONFIG_STACKPROTECTOR`
- `CONFIG_STACKPROTECTOR_STRONG`
The Kernel Address Sanitizer (KASAN) utility also helps in identifying
some stack corruption issues.
## Function tracer
Function tracer (ftrace) provides tracing utilities to perform system-wide profiling and tracing at runtime.
To use ftrace, enable the following configuration options:
- `CONFIG_FTRACE`
- `CONFIG_HAVE_FUNCTION_TRACER`
- `CONFIG_HAVE_FUNCTION_GRAPH_TRACER`
- `CONFIG_HAVE_DYNAMIC_FTRACE`
- `CONFIG_HAVE_FTRACE_MCOUNT_RECORD`
The following are some operations that can be performed using ftrace to debug kernel issues:
### Dump ftrace information to kmsg buffer
To dump the ftrace information into the kmsg buffer anytime from the source code, call the `ftrace_dump(DUMP_ALL)` function.
To increase the buffer size of the ftrace ring, run the following command:
echo 200 > /sys/kernel/debug/tracing/buffer_size_kb
Copy to clipboard
### Enable work queue trace
To enable work queue tracing, run the following commands:
mount -t debugfs none /sys/kernel/debug
Copy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/workqueue/enable
Copy to clipboard
echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
Copy to clipboard
cat /sys/kernel/debug/tracing/trace_pipe
Copy to clipboard
cat /sys/kernel/debug/tracing/per_cpu/cpu1/trace
Copy to clipboard
Sample output:
# tracer: nop
#
# entries-in-buffer/entries-written: 8682/8682 #P:1
#
# \_-----=> irqs-off
# / \_----=> need-resched
# | / \_---=> hardirq/softirq
# || / \_--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<...>-4783 [001] d.s4 7524.354249: workqueue\_queue\_work: work struct=f2d91ee4 function=free\_css\_set\_work workqueue=f6427d80 req\_cpu=1 cpu=1
<idle>-0 [001] d.h4 7524.424196: workqueue\_queue\_work: work struct=c10c32a8 function=def\_work\_fn workqueue=f55c7880 req\_cpu=1 cpu=4
e.process.gapps-4758 [001] dNs4 7524.454227: workqueue\_queue\_work: work struct=c4727fa4 function=free\_css\_set\_work workqueue=f6427d80 req\_cpu=1 cpu=1
Binder\_D-1693 [001] d.s3 7524.504198: workqueue\_queue\_work: work struct=cec6275c function=do\_dbs\_timer workqueue=f5424680 req\_cpu=1 cpu=1
<...>-4832 [001] d.h3 7524.574194: workqueue\_queue\_work: work struct=c10c32a8 function=def\_work\_fn workqueue=f55c7880 req\_cpu=1 cpu=4
### Enable Interrupt request (IRQ) trace
To enable interrupt tracing, run the following commands:
mount -t debugfs none /sys/kernel/debug
Copy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/irq/irq_handler_entry/enable
Copy to clipboard
cat /sys/kernel/debug/tracing/trace
Copy to clipboard
cat /sys/kernel/debug/tracing/trace_pipe
Copy to clipboard
Sample output:
TASK-PID CPU# TIMESTAMP FUNCTION
| | | | |
adbd-302 [000] 295.075568: irq\_handler\_entry: irq=132 name=msm\_otg
adbd-302 [000] 295.075599: irq\_handler\_entry: irq=132 name=msm\_hsusb
adbd-302 [000] 295.075782: irq\_handler\_entry: irq=132 name=msm\_otg
adbd-302 [000] 295.075782: irq\_handler\_entry: irq=132 name=msm\_hsusb
<idle>-0 [000] 295.076270: irq\_handler\_entry: irq=132 name=msm\_otg
<idle>-0 [000] 295.076270: irq\_handler\_entry: irq=132 name=msm\_hsusb
<idle>-0 [000] 295.077155: irq\_handler\_entry: irq=18 name=gp\_timer
<idle>-0 [000] 295.087166: irq\_handler\_entry: irq=18 name=gp\_timer
<idle>-0 [000] 295.097146: irq\_handler\_entry: irq=18 name=gp\_timer
### Enable scheduler trace
To trace context switches between tasks, run the following commands to
enable the `sched_switch` tracer:
mount -t debugfs none /sys/kernel/debug
Copy to clipboard
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
Copy to clipboard
cat /sys/kernel/debug/tracing/trace
Copy to clipboard
cat /sys/kernel/debug/tracing/trace_pipe
Copy to clipboard
Sample output:
TASK-PID CPU# TIMESTAMP FUNCTION
| | | | |
WindowManagerPo-484 [000] 994.583135: sched\_switch: prev\_comm=WindowManagerPo
SurfaceFlinger-334 [000] 994.583652: sched\_switch: prev\_comm=SurfaceFlinger
WindowManagerPo-484 [000] 994.584320: sched\_switch: prev\_comm=WindowManagerPo
GL updater-675 [000] 994.584370: sched\_switch: prev\_comm=GL updater
WindowManagerPo-484 [000] 994.584424: sched\_switch: prev\_comm=WindowManagerPo
### Find IRQ latency
To find the maximum IRQ latency and pre-emption latency in the system, enable ftrace configurations for `IRQOFF` and `PREEMPTIONOFF` as follows:
CONFIG_IRQSOFF_TRACER =Y
CONFIG_PREEMPT_TRACER =Y
Copy to clipboard
For this configuration to take effect, recompile the kernel. This configuration detects latency in milliseconds effectively.
To enable tracing, run the following commands:
echo 0 > /sys/kernel/debug/tracing/tracing_enabled
Copy to clipboard
echo 0 > /sys/kernel/debug/tracing/tracing_max_latency
Copy to clipboard
echo irqsoff > /sys/kernel/debug/tracing/current_tracer
Copy to clipboard
echo 1 > /sys/kernel/debug/tracing/tracing_enabled
Copy to clipboard
cat /sys/kernel/debug/tracing/trace
Copy to clipboard
To find the maximum latency observed in the system, configure `tracing_max_latency = 0`. To detect any latency higher than the specified limit, set the threshold level in microseconds. For example,
echo 2000 > /sys/kernel/debug/tracing/tracing_thresh
Copy to clipboard
The following example shows the traces for IRQ latency of 16 ms:
cat /sys/kernel/debug/tracing/trace
# tracer: irqsoff
#
# WARNING: FUNCTION TRACING IS CORRUPTED
# MAY BE MISSING FUNCTION EVENTS
# irqsoff latency trace v1.1.5 on 3.4.0-perf-g7736d93-dirty
# --------------------------------------------------------------------
# latency: 16757 us, #4/4, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:1)
# -----------------
# | task: EventThread-534 (uid:1000 nice:-9 policy:0 rt\_prio:0)
# -----------------
# => started at: \_raw\_spin\_lock\_irqsave
# => ended at: \_raw\_spin\_unlock\_irqrestore
#
#
# \_------=> CPU#
# / \_-----=> irqs-off
# | / \_----=> need-resched
# || / \_---=> hardirq/softirq
# ||| / \_--=> preempt-depth
# |||| / delay
# cmd pid ||||| time | caller
# / ||||| | /
<...>-534 0d... 0us!: \_raw\_spin\_lock\_irqsave
<...>-534 0d..1 16756us+: \_raw\_spin\_unlock\_irqrestore
<...>-534 0d..1 16758us+: trace\_hardirqs\_on <-\_raw\_spin\_unlock\_irqrestore
<...>-534 0d..1 16780us : <stack trace>
=> trace\_hardirqs\_on
=> \_raw\_spin\_unlock\_irqrestore
=> clk\_enable
=> mdss\_dsi\_clk\_enable
=> mdss\_dsi\_clk\_ctrl
=> mdss\_dsi\_clk\_req
=> mdss\_dsi\_event\_handler
=> mdss\_mdp\_ctl\_intf\_event
=> mdss\_mdp\_cmd\_add\_vsync\_handler
=> mdss\_mdp\_overlay\_vsync\_ctrl
=> mdss\_mdp\_overlay\_ioctl\_handler
=> mdss\_fb\_ioctl
=> do\_fb\_ioctl
=> fb\_ioctl
=> do\_vfs\_ioctl
=> sys\_ioctl
=> ret\_fast\_syscall
/sys/kernel/debug/tracing #
For more information, see `Documentation/trace/ftrace.txt` available at [https://www.kernel.org/doc/Documentation](https://www.kernel.org/doc/Documentation).
## Collect and parse RAM dump
A RAM dump is a snapshot of the system memory at the time of device failure and is useful for debugging various crash issues.
### Enable RAM dump
By default, the RAM dump is enabled in the `debug` build, but it is disabled in the `perf`/non-DEBUG build using the kernel command-line parameter. To configure the RAM dump, use the `qcom_scm.download_mode` parameter in the `meta-qcom-hwe/conf/machine/include/` file as:
- 0 – Disable
- 1 – Enable
The following table lists the `.inc` file corresponding to different chipsets:
| Chipset | Filename |
| --- | --- |
| QCS5430/QCS6490 | `meta-qcom-hwe/conf/machine/include/qcom-qcs6490.inc` |
| QCS8275 | `meta-qcom-hwe/conf/machine/include/qcom-qcs8300.inc` |
| QCS9075 | `meta-qcom-hwe/conf/machine/include/qcom-qcs9100.inc` |
### Collect RAM dump
To collect the RAM dump over USB, use the Product Configuration Assistant Tool (PCAT). Download the PCAT tool from the [Qualcomm Software Center](https://softwarecenter.qualcomm.com/#/catalog/item/PCAT). After installing PCAT, access the PCAT user guide at the following locations:
- Windows host: `C:\Program Files(x86)\Qualcomm\PCAT\Docs`
- Linux host: `/opt/qcom/PCAT/Docs/`
To collect RAM dump using PCAT, do the following:
1. Start PCAT on a Windows host and connect the device, which is already in the RAM dump mode.
If necessary, to trigger a device crash, click **CRASH** on the PCAT interface.
After the device is connected to the Windows host, PCAT automatically starts capturing the RAM dump and dump collection progress is displayed on the PCAT UI.
2. To view the dump, click **OPEN**.
The following is an example showing the contents of a RAM dump directory:
Note
The Linux host uses a command-line interface, instead of the GUI for the PCAT-based RAM dump capture, such as
`PCAT -PLUGIN CC -DEVICE -DUMPDIR /tmp -RESET TRUE -UNIQUETS TRUE`.
## Parse RAM dump using RAMParser
Linux RAM dump parser (RAMParser) is an open-source tool used to parse RAM dump on Qualcomm Linux devices. The RAMParser processes the RAM dump using the Linux kernel symbol file, which includes vmlinux and kernel object modules, and extracts useful information such as process stacks, IRQ, and workqueues.
Note
The RAMParser tool is built and verified only for Windows.
### Prerequisites
- RAM dump and the corresponding `vmlinux` file
- Software images and scripts
- Windows PC
- Python 3.7 or a later version
python -m pip install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org prettytable
Copy to clipboard
python -m pip install --trusted-host files.pythonhosted.org --trusted-host pypi.org --trusted-host pypi.python.org pyelftools
Copy to clipboard
- Pyelftools package
1. Download Pyelftools package from [https://github.com/eliben/pyelftools](https://github.com/eliben/pyelftools).
2. Unzip the downloaded file and look for `pyelftools-master>elftools` directory.
3. Copy elftools directory to <installed Python `path>\Lib\site-packages`.
- RAMParser software
Download the RAMParser software from [https://git.codelinaro.org/clo/la/platform/vendor/qcom-opensource/tools/-/tree/opensource-tools.lnx.1.0](https://git.codelinaro.org/clo/la/platform/vendor/qcom-opensource/tools/-/tree/opensource-tools.lnx.1.0).
Note
The RAMParser software must always be present in the C drive on a Windows host.
- (Optional) TRACE32 software 2023.12 or a later version on C drive (`C:\T32`)
The TRACE32 is used to load the RAM dump in the TRACE32 simulator using the RAMParser output files.
### Set up toolchains
The RAMParser requires access to gdb and nm tools. You can specify the paths to the gdb and nm tools in one of the following ways:
- Using `--gdb-path` and `--nm-path` to specify the absolute path
- Using `CROSS_COMPILE` to specify the prefix
- Using `local_settings.py` file
Note
Availability of gdb or nm only on the path is not supported because there are too many variations on the names to invoke.
To set up toolchains using the `local_settings.py` file, do the following:
1. Create a directory named `ramparser_utils\utils` in the root directory.
2. Download the toolchain in the `ramparser_utils\utils` directory from [https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads](https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads).
### local\_settings.py
The RAMParser attempts to determine most of the settings automatically. However, there are some settings that are unique to the runtime environment. These unique settings are specified in the
`local_settings.py` file. Because the `local_settings.py` is a Python file, it can take advantage of the Python features.
Note
For RAMParser to pick the correct path for toolchain utilities, add the `local_settings.py` file to the path `/tools/linux-ramdump-parser-v2` directory.
The format of the `local_settings.py` file is:
=
Copy to clipboard
The `local_settings.py` file supports the following features:
Table : Supported features in local_settings.py file
| Feature | Description |
| --- | --- |
| gdb\_path | Absolute path to the gdb tool for the RAM dump |
| nm\_path | Absolute path to the nm tool for the RAM dump |
| gdb64\_path | Absolute path to the 64‑bit gdb tool for the RAM dump |
| nm64\_path | Absolute path to the 64‑bit nm tool for the RAM dump |
| objdump\_path | Absolute path to object dump tool for the RAM dump |
| objdump64\_path | Absolute path to the 64‑bit object dump tool for the RAM dump |
### Example: local\_settings.py
import parser_util,os,sys
path = os.path.abspath(os.path.dirname(__file__))
ramparser_utils_path = os.path.abspath(os.path.join(path, "../../ramparser_utils"))
nm_path = ramparser_utils_path + "\\utils\\arm-none-eabi-nm.exe"
gdb_path = ramparser_utils_path + "\\utils\\arm-none-eabi-gdb.exe"
objdump_path = ramparser_utils_path + "\\utils\\objdump.exe"
nm64_path = ramparser_utils_path + "\\utils\\aarch64-linux-gnu-nm.exe"
gdb64_path = ramparser_utils_path + "\\utils\\mingw64\\bin\\gdb-multiarch.exe"
objdump64_path = ramparser_utils_path + "\\utils\\all-objdump.exe"
Copy to clipboard
### RAMParser commands
To parse dumps using RAMParser, run the following command in the Windows Shell:
python ramparse.py --vmlinux --auto-dump --force-hardware --mod_path -o