# Debug Linux user space
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html)
The following open-source tools, dumps, and logs can be used to debug issues related to
Linux user space applications.
The supported open-source debug tools are enabled using the
packagegroup-core-tools-debug.bbappend BitBake file. This
BitBake file is available at:
layers\meta-qcom-hwe\recipes-devtools\packagegroups\. The
procedures to enable and use the open-source tools like ltrace, Valgrind are described
here.
## Debug using ltrace
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html)
By default, ltrace is enabled in the build.
**Prerequisites:**
To use SSH, you must enable SELinux Permissive mode by following the steps mentioned
[here](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-254/how_to.html#how-to-ssh-).
To debug with ltrace using SSH, run the following commands:
1. Run the process status command:
ps -efCopy to clipboard
The list of all
the running processes in the system is displayed.
339 rpc 0:00 /usr/sbin/rpcbind -w -f
345 root 0:00 /lib/systemd/systemd-journald
373 root 0:00 [kworker/3:6-mm_]Copy to clipboard
2. Identify the process id (PID) of the process that has to be debugged.
3. Run the `ltrace`
command:
ltrace -p 345Copy to clipboard
In
this example, `345` is the PID.
The following is the
sample output of the ltrace command:
journal_file_close(0x25e30170, -1, 0, 0
sockaddr_un_unlink(0x25e30170, -1, 0, 0
sd_journal_close(0x25e31c00, 1, 3, 0x409e70) = 0
sd_journal_close(0x25e31d50, 1, 0, 1) = 0
<... sockaddr_un_unlink resumed> ) = 1
<... journal_file_close resumed> ) = 1
journal_file_close(0x25e30170, -1, 0, 0
sockaddr_un_unlink(0x25e30170, -1, 0, 0
sd_event_now(15, 0x41ed20, 10, 64) = 10
openat64(2, 0x41ed20, 10, 64
clock_gettime(2, 0x41ed20, 10, 64) = 6Copy to clipboard
For more information, see [https://man7.org/linux/man-pages/man1/ltrace.1.html](https://man7.org/linux/man-pages/man1/ltrace.1.html).
## Debug using Valgrind
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html)
The Valgrind tool can detect many memory-related errors that are common in C and
C++ programs and lead to crashes and unpredictable behavior such as memory
leaks and corruptions.
By default, the Valgrind tool is enabled in the build. To push debug symbols on
the device, see [Configure debug symbols](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html#debug_symbols).
Use the following command to run the Valgrind tool:
valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=20 --track-fds=yes /usr/bin/tqftpservCopy to clipboard
Sample output:
==622== Memcheck, a memory error detector
==622== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==622== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==622== Command: /usr/bin/tqftpserv
==622==
^C==622==
==622== Process terminating with default action of signal 2 (SIGINT)
==622== at 0x498EF38: select (select.c:69)
==622== by 0x4013CB: ??? (in /usr/bin/tqftpserv)
==622== by 0x48DB1AF: (below main) (libc_start_call_main.h:58)
==622==
==622== FILE DESCRIPTORS: 4 open (3 std) at exit.
==622== Open pf-42 socket 3:
==622== at 0x49984CC: socket (syscall-template.S:120)
==622== by 0x4891293: qrtr_open (in /usr/lib/libqrtr.so.1.0)
==622== by 0x40129B: ??? (in /usr/bin/tqftpserv)
==622== by 0x48DB1AF: (below main) (libc_start_call_main.h:58)
==622==
==622==
==622== HEAP SUMMARY:
==622== in use at exit: 0 bytes in 0 blocks
==622== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==622==
==622== All heap blocks were freed -- no leaks are possible
==622==
==622== For lists of detected and suppressed errors, rerun with: -s
==622== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==622== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)Copy to clipboard
For more information, see [https://valgrind.org/](https://valgrind.org/).
## Debug using GDB
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html)
By default, GDB is not enabled in the build. To enable GDB, do the following:
1. Navigate to the
layers\meta-qcom-hwe\recipes-devtools\packagegroups\
folder and open the packagegroup-core-tools-debug.bbappend
file.
2. Modify the packagegroup-core-tools-debug.bbappend file to
add `gdb` in the package list. Skip this step if the package name
exists in the package list.
3. Recompile and flash the build on the device.
For efficient debugging with the GDB, debug symbols must be enabled and to verify the
same run the following commands on the host Linux machine:
readelf --debug-dump=decodedline Copy to clipboard
Or
objdump --syms | grep -i ‘debug’Copy to clipboard
If the debug symbols are not enabled, you can compile all the required executables or
shared libraries with the `-g` CFLAG. To push debug symbols on the
device, see [Configure debug symbols](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html#debug_symbols).
The following is an example GDB command:
gdb --pid 502Copy to clipboard
**Sample output:**
gdb: warning: Couldn't determine a path for the index cache directory.
GNU gdb (GDB) 11.2
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-qcom-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 502
Reading symbols from /usr/bin/tqftpserv...
(No debugging symbols found in /usr/bin/tqftpserv)
Reading symbols from /usr/lib/libqrtr.so.1...
(No debugging symbols found in /usr/lib/libqrtr.so.1)
Reading symbols from /lib/libc.so.6...
Reading symbols from /lib/.debug/libc.so.6...
Reading symbols from /lib/ld-linux-aarch64.so.1...
Reading symbols from /lib/.debug/ld-linux-aarch64.so.1...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
0x0000ffffab00ef34 in __GI___select (nfds=4, readfds=0xffffc2ec14f8, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69
69 ../sysdeps/unix/sysv/linux/select.c: No such file or directory.
(gdb)
(gdb)
(gdb)
(gdb)
Copy to clipboard
Some of the useful GDB commands are:
- `(gdb) bt`: provides a backtrace of the current thread.
- `(gdb) info threads`: lists the IDs of currently known
threads.
- `(gdb) thread 2`: switches to thread 2.
- `(gdb) where`: shows the current line number and the function you
are in.
- `(gdb) thread apply all bt full`: provides a backtrace of all
threads.
- `(gdb) info sharedlibrary`: lists the names of shared libraries
used.
- `(gdb) info reg`: lists the CPU registers.
For more information, see [https://man7.org/linux/man-pages/man1/gdb.1.html](https://man7.org/linux/man-pages/man1/gdb.1.html).
## Collect core dump
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html)
A core dump is a memory snapshot of a user space process. You can use core dump to
analyze the cause of a process crash.
Core dump is collected per Yocto Linux standard for all user space process crashes. By
default, core dump is enabled in the build. If core dump is not enabled, you can use the
following patch to enable the core dump
layers\meta-qcom-distro\recipes-products\packagegroups\packagegroup-qcom.bb.
RDEPENDS:packagegroup-support-utils = "\
chrony \
libinput \
libinput-bin \
libnl \
libxml2 \
+++ procps \
"Copy to clipboard
Note: When enabling core dump using this patch, you must rebuild
and reflash the device.
To push debug symbols on the device, see [Configure debug symbols](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html#debug_symbols).
By default, the core dump is saved on the device at /var/coredump.
To verify the location of the core dump, run the following
command:
cat /proc/sys/kernel/core_patternCopy to clipboard
Sample
output:
/var/coredump/%e.coreCopy to clipboard
The size of the core dump should be more than 0 (zero). To verify the size of the core
dump, run the following
command:
ulimit -cCopy to clipboard
The expected output is: unlimited.
### Using GDB on core dump
You can force kill a process to get the core dump. Following are the example
commands:
ps -ef | grep -i 'tqftp*'Copy to clipboard
Sample
output:
root 1024 1 0 17:46 ? 00:00:00 /usr/bin/tqftpserv
root 1047 934 0 17:47 pts/0 00:00:00 grep -i tqftp*Copy to clipboard
kill -11 1024Copy to clipboard
cd /var/coredumpCopy to clipboard
lsCopy to clipboard
Sample
output:
tqftpserv.coreCopy to clipboard
The tqftpserv.core is the core dump file.
If the debug symbols are already pushed to the device, you can use the GDB tool on
the core dump with the following
commands:
cd /usr/bin/Copy to clipboard
gdb tqftpserv /var/coredump/tqftpserv.coreCopy to clipboard
Sample
output:
gdb: warning: Couldn't determine a path for the index cache directory.
GNU gdb (GDB) 11.2
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
Type "apropos word" to search for commands related to "word"...
Reading symbols from tqftpserv...
warning: exec file is newer than core file.
[New LWP 1024]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `/usr/bin/tqftpserv'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000ffff8629ef34 in __GI___select (nfds=4,
readfds=readfds@entry=0xffffd04ccb28, writefds=writefds@entry=0x0,
--Type for more, q to quit, c to continue without paging--
exceptfds=exceptfds@entry=0x0, timeout=timeout@entry=0x0)
at ../sysdeps/unix/sysv/linux/select.c:69
69 ../sysdeps/unix/sysv/linux/select.c: No such file or directory.
(gdb) bt
#0 0x0000ffff8629ef34 in __GI___select (nfds=4,
readfds=readfds@entry=0xffffd04ccb28, writefds=writefds@entry=0x0,
exceptfds=exceptfds@entry=0x0, timeout=timeout@entry=0x0)
at ../sysdeps/unix/sysv/linux/select.c:69
#1 0x00000000004013cc in main (argc=, argv=)
at tqftpserv.c:552
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)Copy to clipboard
For more information, see [https://man7.org/linux/man-pages/man5/core.5.html](https://man7.org/linux/man-pages/man5/core.5.html).
## Debug using gdbserver
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html)
gdbserver tool lets you remotely run the GDB tool from the host machine. The gdbserver is
helpful in debugging when there is a space constraint for debug symbols on the device.
### Enable gdbserver
By default, gdbserver is not enabled in the build. To enable the gdbserver, do the
following:
1. Navigate to the
layers\meta-qcom-hwe\recipes-devtools\packagegroups\
folder and open the
packagegroup-core-tools-debug.bbappend file.
2. Modify the packagegroup-core-tools-debug.bbappend file
to add the `gdbserver` in the package list.
3. Recompile and flash the build on the device.
### Configure gdbserver on device
To configure the gdbserver tool on the device, run the following commands using
SSH:
setenforce 0Copy to clipboard
mount -o rw,remount /Copy to clipboard
gdbserver :8888
Copy to clipboard
For example,
gdbserver :8888 /usr/bin/tqftpservCopy to clipboard
Sample output:
The sample output indicates that the device is ready to communicate with the
host.
### Configure gdb on a Linux host machine
To configure the gdb tool on a Linux host machine, do the following using SSH:
1. Install gdb,
gdb-multiarch.
sudo apt-get install gdb gdb-multiarchCopy to clipboard
2. Create a debug folder to capture all symbols from the
build.
mkdir test_gdbserverCopy to clipboard
3. Copy the rootfs having the debug symbols from the build location to the
debug folder
test\_gdbserver.
cp -rf \build-qcom-wayland\tmp-glibc\deploy\images\qcm6490\qcom-multimedia-image-qcm6490-dbg.rootfs.tar.bz2 test_gdbserverCopy to clipboard
cp -rf \build-qcom-wayland\tmp-glibc\deploy\images\qcm6490\qcom-multimedia-image-qcm6490.rootfs.tar.xz test_gdbserverCopy to clipboard
cd test_gdbserverCopy to clipboard
tar -xvf qcom-multimedia-image-qcm6490-dbg.rootfs.tar.bz2Copy to clipboard
tar -xvf qcom-multimedia-image-qcm6490.rootfs.tar.xzCopy to clipboard
The
test\_gdbserver is now ready with the
symbols.
4. Start the gdb-multiarch tool with the command `gdb-multiarch
`.
For example,
gdb-multiarch test_gdbserver/usr/bin/tqftpservCopy to clipboard
The
gdb-multiarch starts on the gdb console.
5. Run the following commands on the gdb
console:
set gnutarget elf64-littleaarch64Copy to clipboard
Set
sysroot
<path-to-new-debug-directory>.
set sysroot \test_gdbserverCopy to clipboard
target remote :8888Copy to clipboard
btCopy to clipboard
bCopy to clipboard
info threadsCopy to clipboard
The following figures show the sample output during the configuration process.
For more information, see [https://man7.org/linux/man-pages/man1/gdbserver.1.html](https://man7.org/linux/man-pages/man1/gdbserver.1.html).
## User space logs
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html)
### Syslog
To verify if syslogs are generated, run the
command:
cat /var/log/user.logCopy to clipboard
If the syslogs are not generated, run the command:
tail -f /var/log/messagesCopy to clipboard
Sample log:
Apr 28 17:42:30 qcm6490 daemon.info avahi-daemon[516]: No service file found in /etc/avahi/services.
Apr 28 17:42:30 qcm6490 daemon.info avahi-daemon[516]: Joining mDNS multicast group on interface lo.IPv6 with address ::1.
Apr 28 17:42:30 qcm6490 daemon.info avahi-daemon[516]: New relevant interface lo.IPv6 for mDNS.
Apr 28 17:42:30 qcm6490 daemon.info avahi-daemon[516]: Joining mDNS multicast group on interface lo.IPv4 with address 127.0.0.1.
Apr 28 17:42:30 qcm6490 daemon.info avahi-daemon[516]: New relevant interface lo.IPv4 for mDNS.Copy to clipboard
### journalctl logs
You can generate the systemd journalctl logs using the following
commands:
journalctl -efCopy to clipboard
Sample log:
Apr 28 17:42:28 qcm6490 kernel: spmi-temp-alarm c440000.spmi:pmic@2:temp-alarm@2400: error -ENXIO: IRQ index 0 not found
Apr 28 17:42:28 qcm6490 kernel: qcom-spmi-adc5 c440000.spmi:pmic@2:adc@3100: Invalid dig version read -19
Apr 28 17:42:28 qcm6490 kernel: qcom-spmi-adc5 c440000.spmi:pmic@2:adc@3100: error -ENODEV: adc get dt data failed
Apr 28 17:42:28 qcm6490 kernel: qcom-spmi-adc5 c440000.spmi:pmic@0:adc@3100: error -EINVAL: adc get dt data failed
Apr 28 17:42:28 qcm6490 kernel: qcom-spmi-adc5: probe of c440000.spmi:pmic@0:adc@3100 failed with error -22
Apr 28 17:42:28 qcm6490 kernel: dwc3 a600000.usb: Adding to iommu group
Apr 28 17:42:29 qcm6490 systemd[1]: First Boot Complete was skipped because of a failed condition check (ConditionFirstBoot=yes).
Apr 28 17:42:29 qcm6490 systemd[1]: Reached target Hardware activated USB gadget.Copy to clipboard
## Configure debug symbols
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html)
Debug symbols are required to parse core dumps. Yocto, by design compiles a package and
splits it into multiple packages. For example, if a hello.cpp file
is compiled by a hello\_0.1.bb file, several packages are generated.
The following table lists the packages that are relevant to the core dump.
| Package | Description |
| --- | --- |
| hello\_0.1-r0\_armv8-2a.ipk | This package contains a stripped executable. This is the only package that is packed in the image for the device. |
| hello-dbg\_0.1-r0\_armv8-2a.ipk | This package contains the debug symbols and is never packed in the image. The debug package (-dbg) bloats the image significantly and causes problems in fitting in the image. Also, apart from debugging this package has no runtime value. Hence, as a strategy Yocto does not pack any -dbg package in the image. |
| hello-dev\_0.1-r0\_armv8-2a.ipk | This package contains the exported headers and exported libraries used by the dependent modules for compilation time. |
As per Yocto standards, the debug symbols are stored at the build location path:
tmp-glibc\deploy\ipk\armv8-2a. For example,
tqftpserv-dbg\_0.0+0+de42697a24-r0\_armv8-2a.ipk package contains
the debug symbols for /usr/bin/tqftpserv directory.
To push the debug symbols on the device, do the following using SSH:
1. Remount the root
partition.
mount -o rw,remount /Copy to clipboard
2. Using `scp` command, push the debug symbols
(tqftpserv-dbg\_0.0+0+de42697a24-r0\_armv8-2a.ipk) to the
device at any available partition like
/data/.
chmod 777 /data/tqftpserv-dbg_0.0+0+de42697a24-r0_armv8-2a.ipkCopy to clipboard
cd dataCopy to clipboard
3. Install debug symbols on the device.
opkg install --nodeps tqftpserv-dbg_0.0+0+de42697a24-r0_armv8-2a.ipkCopy to clipboard
After pushing the debug symbols to the device, the symbols are saved at the same path of
the executable directory in the .debug directory. For example, for
/usr/bin/tqftpserv directory, the debug symbols are saved at
/usr/bin/.debug directory.
To know the debug symbols that are available on the device, run the following
commands:
cd /usr/bin/.debugCopy to clipboard
lsCopy to clipboard
Sample
output:
gencat getent locale pcprofiledump sprof zdump
getconf iconv makedb pldd tqftpservCopy to clipboard
## Reboot commands
Source: [https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html](https://docs.qualcomm.com/doc/80-70014-12/topic/using_open_source_debug_tools.html)
The user space reboot framework triggers the reboot using the kernel driver
(`drivers/firmware/psci/psci.c`). Following are the sample reboot
commands that can be run using SSH:
- `reboot edl`: This command reboots the device to the Emergency
download (EDL) mode. For more information on EDL mode, see [Qualcomm Linux Build Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-254/flash_images.html#emergency-download-edl-mode).
- `reboot bootloader`: This command reboots the device to Fastboot
mode. For more information on Fastboot mode, see [Qualcomm Linux Boot Guide](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-4/fastboot.html).
Following are the optional parameters available in the `reboot` command:
Last Published: Jul 12, 2024
[Previous Topic
Overview](https://docs.qualcomm.com/bundle/publicresource/80-70014-12/topics/Debug-overview.md) [Next Topic
Debug Linux kernel space](https://docs.qualcomm.com/bundle/publicresource/80-70014-12/topics/debugging_linux_kernel.md)