# Debug Linux user space issues
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html)
The following figure illustrates the open-source tools, dumps, and logs that can be used
to debug issues related to Linux user space applications.
Figure : Resources to debug issues in the Linux user space
The supported open-source debug tools are enabled using the
packagegroup-core-tools-debug.bbappend BitBake file. This
BitBake file is available at:
layers/meta-qcom-hwe/recipes-devtools/packagegroups/. The
procedures to enable and use the open-source tools such as ltrace are described
here.
## Debug using ltrace
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html)
By default, ltrace is enabled in the build.
**Prerequisite**
Set up SSH, follow the steps mentioned [here](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-254/how_to.html#use-ssh).
**Procedure**
To debug with ltrace, run the following commands using SSH:
1. Run the process status command:
ps -efCopy to clipboard
The
list of all the running processes in the system is
displayed.
339 rpc 0:00 /usr/sbin/rpcbind -w -f
345 root 0:00 /lib/systemd/systemd-journald
373 root 0:00 [kworker/3:6-mm_]Copy to clipboard
2. Identify the process id (PID) of the process that you want to debug.
3. Run the `ltrace`
command:
ltrace -p 345Copy to clipboard
In
this example, `345` is the PID.
Sample output:
journal_file_close(0x25e30170, -1, 0, 0
sockaddr_un_unlink(0x25e30170, -1, 0, 0
sd_journal_close(0x25e31c00, 1, 3, 0x409e70) = 0
sd_journal_close(0x25e31d50, 1, 0, 1) = 0
<... sockaddr_un_unlink resumed> ) = 1
<... journal_file_close resumed> ) = 1
journal_file_close(0x25e30170, -1, 0, 0
sockaddr_un_unlink(0x25e30170, -1, 0, 0
sd_event_now(15, 0x41ed20, 10, 64) = 10
openat64(2, 0x41ed20, 10, 64
clock_gettime(2, 0x41ed20, 10, 64) = 6Copy to clipboard
For more information, see [https://man7.org/linux/man-pages/man1/ltrace.1.html](https://man7.org/linux/man-pages/man1/ltrace.1.html).
## Debug using Valgrind
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html)
The Valgrind tool can detect memory-related errors that are common in C and C++
programs. These errors cause crashes and unpredictable behavior such as
memory leaks and memory corruptions. By default, the Valgrind tool is
enabled in the build.
**Prerequisite**
Set up SSH, follow the steps mentioned [here](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-254/how_to.html#use-ssh).
**Procedure**
To use the Valgrind tool, do the following:
1. Push debug symbols to the device. For instructions, see [Configure debug symbols](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html#debug_symbols).
2. Run the Valgrind tool on the
device.
valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=20 --track-fds=yes /usr/bin/tqftpservCopy to clipboard
Sample output:
==622== Memcheck, a memory error detector
==622== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==622== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==622== Command: /usr/bin/tqftpserv
==622==
^C==622==
==622== Process terminating with default action of signal 2 (SIGINT)
==622== at 0x498EF38: select (select.c:69)
==622== by 0x4013CB: ??? (in /usr/bin/tqftpserv)
==622== by 0x48DB1AF: (below main) (libc_start_call_main.h:58)
==622==
==622== FILE DESCRIPTORS: 4 open (3 std) at exit.
==622== Open pf-42 socket 3:
==622== at 0x49984CC: socket (syscall-template.S:120)
==622== by 0x4891293: qrtr_open (in /usr/lib/libqrtr.so.1.0)
==622== by 0x40129B: ??? (in /usr/bin/tqftpserv)
==622== by 0x48DB1AF: (below main) (libc_start_call_main.h:58)
==622==
==622==
==622== HEAP SUMMARY:
==622== in use at exit: 0 bytes in 0 blocks
==622== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==622==
==622== All heap blocks were freed -- no leaks are possible
==622==
==622== For lists of detected and suppressed errors, rerun with: -s
==622== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==622== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)Copy to clipboard
For more information, see [https://valgrind.org/](https://valgrind.org/).
## Debug using GDB
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html)
By default, the GNU debugger (GDB) is not enabled in the build. To enable GDB, do the
following on the host machine:
1. Navigate to the
layers/meta-qcom-hwe/recipes-devtools/packagegroups/
directory and open the
packagegroup-core-tools-debug.bbappend file.
2. Verify that the debug symbols are
enabled.
readelf --debug-dump=decodedline Copy to clipboard
Or
objdump --syms | grep -i ‘debug’Copy to clipboard
If
the debug symbols are not enabled, compile all the required executables or
shared libraries with the `-g` CFLAG. To push debug symbols
to the device, see [Configure debug symbols](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html#debug_symbols).
3. Modify the packagegroup-core-tools-debug.bbappend file to
add `gdb` in the package list. Skip this step if the package name
is already in the package list.
4. Recompile and flash the build on the device.
To debug using GDB, run the following command on the
device:
gdb --pid 502Copy to clipboard
Sample output:
gdb: warning: Couldn't determine a path for the index cache directory.
GNU gdb (GDB) 11.2
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-qcom-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 502
Reading symbols from /usr/bin/tqftpserv...
(No debugging symbols found in /usr/bin/tqftpserv)
Reading symbols from /usr/lib/libqrtr.so.1...
(No debugging symbols found in /usr/lib/libqrtr.so.1)
Reading symbols from /lib/libc.so.6...
Reading symbols from /lib/.debug/libc.so.6...
Reading symbols from /lib/ld-linux-aarch64.so.1...
Reading symbols from /lib/.debug/ld-linux-aarch64.so.1...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
0x0000ffffab00ef34 in __GI___select (nfds=4, readfds=0xffffc2ec14f8, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69
69 ../sysdeps/unix/sysv/linux/select.c: No such file or directory.
(gdb)
(gdb)
(gdb)
(gdb)
Copy to clipboard
The following are some of the commonly used GDB commands:
Table : Common GDB commands
| Command | Description |
| --- | --- |
| `(gdb) bt` | This command provides a backtrace of the current thread. |
| `(gdb) info threads` | This command lists the IDs of currently known threads. |
| `(gdb) thread 2` | This command switches to thread 2. |
| `(gdb) where` | This command shows the current line number and the function that you are in. |
| `(gdb) thread apply all bt full` | This command provides a backtrace of all threads. |
| `(gdb) info sharedlibrary` | This command lists the names of shared libraries used. |
| `(gdb) info reg` | This command lists the CPU registers. |
For more information, see [https://man7.org/linux/man-pages/man1/gdb.1.html](https://man7.org/linux/man-pages/man1/gdb.1.html).
## Collect core dump
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html)
A core dump is a memory snapshot of a user space process that can be used to analyze the
cause of a process crash.
The core dump is collected per Yocto Linux standard for all user space process crashes.
By default, the core dump is enabled in the build. The generated core dump is saved on
the device at /var/coredump.
To verify the location of the core dump, run the following
command:
cat /proc/sys/kernel/core_patternCopy to clipboard
Sample
output: /var/coredump/%e.core
The size of the core dump must be more than 0 (zero). To verify the size of the core
dump, run the following
command:
ulimit -cCopy to clipboard
The expected output is: unlimited.
If the core dump is not enabled, use the following file to enable it:
layers/meta-qcom-distro/recipes-products/packagegroups/packagegroup-qcom.bb.
RDEPENDS:packagegroup-support-utils = "\
chrony \
libinput \
libinput-bin \
libnl \
libxml2 \
+++ procps \
"Copy to clipboard
Note: While enabling core dump using this patch, you must rebuild
and reflash the device.
### Using GDB on core dump
To generate a core dump, forcibly kill a process. The following are the example
commands:
ps -ef | grep -i 'tqftp*'Copy to clipboard
Sample
output:
root 1024 1 0 17:46 ? 00:00:00 /usr/bin/tqftpserv
root 1047 934 0 17:47 pts/0 00:00:00 grep -i tqftp*Copy to clipboard
kill -11 1024Copy to clipboard
cd /var/coredumpCopy to clipboard
lsCopy to clipboard
Sample
output: tqftpserv.core
The tqftpserv.core is the core dump file.
If the debug symbols are already pushed to the device, run the following commands to
use the GDB tool on the core
dump:
cd /usr/bin/Copy to clipboard
gdb tqftpserv /var/coredump/tqftpserv.coreCopy to clipboard
Sample
output:
gdb: warning: Couldn't determine a path for the index cache directory.
GNU gdb (GDB) 11.2
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
Type "apropos word" to search for commands related to "word"...
Reading symbols from tqftpserv...
warning: exec file is newer than core file.
[New LWP 1024]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `/usr/bin/tqftpserv'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000ffff8629ef34 in __GI___select (nfds=4,
readfds=readfds@entry=0xffffd04ccb28, writefds=writefds@entry=0x0,
--Type for more, q to quit, c to continue without paging--
exceptfds=exceptfds@entry=0x0, timeout=timeout@entry=0x0)
at ../sysdeps/unix/sysv/linux/select.c:69
69 ../sysdeps/unix/sysv/linux/select.c: No such file or directory.
(gdb) bt
#0 0x0000ffff8629ef34 in __GI___select (nfds=4,
readfds=readfds@entry=0xffffd04ccb28, writefds=writefds@entry=0x0,
exceptfds=exceptfds@entry=0x0, timeout=timeout@entry=0x0)
at ../sysdeps/unix/sysv/linux/select.c:69
#1 0x00000000004013cc in main (argc=, argv=)
at tqftpserv.c:552
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)Copy to clipboard
For more information, see [https://man7.org/linux/man-pages/man5/core.5.html](https://man7.org/linux/man-pages/man5/core.5.html).
## Debug using gdbserver
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html)
The gdbserver utility allows you to run the GDB tool remotely from the host machine. This
utility is helpful in debugging issues when there is a storage constraint for debug
symbols on the device.
### Enable gdbserver on device
By default, gdbserver is not enabled in the build. To enable the gdbserver, do the
following:
1. Navigate to the
layers/meta-qcom-hwe/recipes-devtools/packagegroups/
directory and open the
packagegroup-core-tools-debug.bbappend file.
2. Modify the packagegroup-core-tools-debug.bbappend file
to add the `gdbserver` in the package list.
3. Recompile and flash the build on the device.
### Configure gdbserver on device
To configure the gdbserver on the device, run the following commands using
SSH:
mount -o rw,remount /Copy to clipboard
gdbserver :8888
Copy to clipboard
For
example:
gdbserver :8888 /usr/bin/tqftpservCopy to clipboard
Sample output:

The sample output indicates that the device is ready to communicate with the
host.
### Configure gdb on a Linux host machine
To configure the gdb tool on a Linux host machine, do the following:
1. Install gdb and gdb-multiarch
tools.
sudo apt-get install gdb gdb-multiarchCopy to clipboard
2. Create a debug directory to capture all the symbols from the
build.
mkdir test_gdbserverCopy to clipboard
3. Copy the rootfs having the debug symbols from the build location to the
debug directory
test\_gdbserver.
cp -f /build-qcom-wayland/tmp-glibc/deploy/images//qcom-multimedia-image--dbg.rootfs.tar.bz2 test_gdbserverCopy to clipboard
cd test_gdbserverCopy to clipboard
tar -xvf qcom-multimedia-image--dbg.rootfs.tar.bz2Copy to clipboard
Note: While running these commands, replace ``
with the appropriate value as mentioned in the following
table.
| Chipset | Value |
| --- | --- |
| QCS6490 | `qcm6490` |
| QCS5430 | `qcm6490` |
| QCS9075 | `qcs9100` |
| | |
The test\_gdbserver is now ready
with the symbols.
4. Start the gdb-multiarch tool using the `gdb-multiarch
` command.
For example,
gdb-multiarch test_gdbserver/usr/bin/tqftpservCopy to clipboard
The
gdb-multiarch tool starts on the gdb console.
5. Run the following commands on the gdb
console:
set gnutarget elf64-littleaarch64Copy to clipboard
Set
sysroot
<path-to-new-debug-directory>.
set sysroot /test_gdbserverCopy to clipboard
target remote :8888Copy to clipboard
btCopy to clipboard
bCopy to clipboard
info threadsCopy to clipboard
The following screenshots show the sample output during the configuration
process:

$ gdb-multiarch /test_gdbserver/usr/bin/.debug/tqftpserv
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.2) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.Copy to clipboard
Reading symbols from /test_gdbserver/usr/bin/.debug/tqftpserv...
(gdb) set gnutarget elf64-littleaarch64
(gdb) set sysroot /test_gdbserver
(gdb) target remote 10.92.168.78:8888
Remote debugging using 10.92.168.78:8888
warning: while parsing target description (at line 71): Vector "v8bf16" references undefined type "bfloat16"
warning: Could not load XML target description; ignoring
warning: Unable to find dynamic linker breakpoint function.
Reading symbols from /test_gdbserver/lib/ld-linux-aarch64.so.1...
(no debugging symbols found)...done.
Reading symbols from /test_gdbserver/lib/.debug/ld-linux-aarch64.so.1...
Loaded symbols for /test_gdbserver/lib/.debug/ld-linux-aarch64.so.1
warning: no loadable sections found in added symbol-file system-supplied DSO at 0xfffff7ffb000
0x0000fffff7fda870 in _start ()
from /test_gdbserver/lib/.debug/ld-linux-aarch64.so.1
(gdb) bt
#0 0x0000fffff7fda870 in _start ()
from /test_gdbserver/lib/.debug/ld-linux-aarch64.so.1
#1 0x0000000000000000 in ?? ()
(gdb) b
Breakpoint 1 at 0xfffff7fda870
(gdb) info threads
Id Target Id Frame
* 1 Thread 3212 0x0000fffff7fda870 in _start ()
(gdb)Copy to clipboard
For more information, see [https://man7.org/linux/man-pages/man1/gdbserver.1.html](https://man7.org/linux/man-pages/man1/gdbserver.1.html).
## Debug with user space logs
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html)
The following logs can be used to debug issues in the user space using SSH on the
device:
### Syslogs
To verify if syslogs are generated, run the following
command:
cat /var/log/user.logCopy to clipboard
If the syslogs are not generated, run the following command:
tail -f /var/log/messagesCopy to clipboard
Sample log:
Apr 28 17:42:30 qcm6490 daemon.info avahi-daemon[516]: No service file found in /etc/avahi/services.
Apr 28 17:42:30 qcm6490 daemon.info avahi-daemon[516]: Joining mDNS multicast group on interface lo.IPv6 with address ::1.
Apr 28 17:42:30 qcm6490 daemon.info avahi-daemon[516]: New relevant interface lo.IPv6 for mDNS.
Apr 28 17:42:30 qcm6490 daemon.info avahi-daemon[516]: Joining mDNS multicast group on interface lo.IPv4 with address 127.0.0.1.
Apr 28 17:42:30 qcm6490 daemon.info avahi-daemon[516]: New relevant interface lo.IPv4 for mDNS.Copy to clipboard
### journalctl logs
To generate the systemd journalctl logs, run the following
command:
journalctl -efCopy to clipboard
Sample log:
Apr 28 17:42:28 qcm6490 kernel: spmi-temp-alarm c440000.spmi:pmic@2:temp-alarm@2400: error -ENXIO: IRQ index 0 not found
Apr 28 17:42:28 qcm6490 kernel: qcom-spmi-adc5 c440000.spmi:pmic@2:adc@3100: Invalid dig version read -19
Apr 28 17:42:28 qcm6490 kernel: qcom-spmi-adc5 c440000.spmi:pmic@2:adc@3100: error -ENODEV: adc get dt data failed
Apr 28 17:42:28 qcm6490 kernel: qcom-spmi-adc5 c440000.spmi:pmic@0:adc@3100: error -EINVAL: adc get dt data failed
Apr 28 17:42:28 qcm6490 kernel: qcom-spmi-adc5: probe of c440000.spmi:pmic@0:adc@3100 failed with error -22
Apr 28 17:42:28 qcm6490 kernel: dwc3 a600000.usb: Adding to iommu group
Apr 28 17:42:29 qcm6490 systemd[1]: First Boot Complete was skipped because of a failed condition check (ConditionFirstBoot=yes).
Apr 28 17:42:29 qcm6490 systemd[1]: Reached target Hardware activated USB gadget.Copy to clipboard
## Configure debug symbols
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html)
Debug symbols are required to parse core dumps. Yocto Linux, by design, compiles a
package and splits it into multiple packages. For example, if
hello\_0.1.bb file compiles hello.cpp file,
several packages are generated.
The following table lists the packages that are relevant to the core dump.
Table : Packages required for parsing core dump
| Package | Description |
| --- | --- |
| hello\_0.1-r0\_armv8-2a.ipk | This package contains a stripped executable. It is the only package that is included in the image for the device. |
| hello-dbg\_0.1-r0\_armv8-2a.ipk | This package contains the debug symbols and is never packed in the image. The debug package (-dbg) increases the image size significantly, which causes problems in including this package in the image. Also, apart from debugging, this package has no runtime value. Therefore, as a strategy, Yocto does not include any -dbg package in the image. |
| hello-dev\_0.1-r0\_armv8-2a.ipk | This package contains the exported headers and libraries that the dependent modules use during compilation. |
According to Yocto Linux standards, the debug symbols are stored at the build location
path: tmp-glibc/deploy/ipk/armv8-2a. For example,
tqftpserv-dbg\_0.0+0+de42697a24-r0\_armv8-2a.ipk package contains
the debug symbols for the /usr/bin/tqftpserv directory.
To push the debug symbols to the device, do the following using SSH:
1. Remount the
rootfs.
mount -o rw,remount /Copy to clipboard
2. Using the `scp` command, push the debug symbols
(tqftpserv-dbg\_0.0+0+de42697a24-r0\_armv8-2a.ipk) to the
device at any available partition such as
/data/.
chmod 777 /data/tqftpserv-dbg_0.0+0+de42697a24-r0_armv8-2a.ipkCopy to clipboard
cd dataCopy to clipboard
3. Install debug symbols on the device.
opkg install --nodeps tqftpserv-dbg_0.0+0+de42697a24-r0_armv8-2a.ipkCopy to clipboard
After pushing the debug symbols to the device, the symbols are saved at the path of the
executable directory in the .debug directory. For example, for
/usr/bin/tqftpserv directory, the debug symbols are saved at
/usr/bin/.debug directory.
To identify the debug symbols that are available on the device, run the following
commands:
cd /usr/bin/.debugCopy to clipboard
lsCopy to clipboard
Sample
output:
gencat getent locale pcprofiledump sprof zdump
getconf iconv makedb pldd tqftpservCopy to clipboard
## Reboot commands
Source: [https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70015-12/topic/using_open_source_debug_tools.html)
The user space reboot framework triggers the reboot using the kernel driver
(`drivers/firmware/psci/psci.c`). The following reboot commands can
be run using SSH:
Table : Reboot commands
| Command | Description |
| --- | --- |
| `reboot edl` | This command reboots the device to the Emergency download (EDL) mode. For more information on EDL mode, see [Qualcomm Linux Build Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-254/flash_images.html#move-to-edl-mode). |
| `reboot bootloader` | This command reboots the device to Fastboot mode. For more information on Fastboot mode, see [Qualcomm Linux Boot Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-4/fastboot.html). |
The following are the optional parameters available to use with the
`reboot` command: 
Last Published: Oct 14, 2024
[Previous Topic
Overview](https://docs.qualcomm.com/bundle/publicresource/80-70015-12/topics/Debug-overview.md) [Next Topic
Debug Linux kernel space issues](https://docs.qualcomm.com/bundle/publicresource/80-70015-12/topics/debugging_linux_kernel.md)