# Debug

Source: [https://docs.qualcomm.com/doc/80-70015-11/topic/debug.html](https://docs.qualcomm.com/doc/80-70015-11/topic/debug.html)

Debug provides a set of common logging and debugging techniques to troubleshoot
        issues in Qualcomm TEE, trusted and client applications, and secure devices.

Note: Run all the SSH commands in
                the SELinux Permissive mode. The Enforcing mode will be supported in the future. For
                instructions on how to connect to the device, see [Qualcomm Linux Build Guide ➝ How to ➝ Use
                    SSH](https://docs.qualcomm.com/bundle/publicresource/topics/80-70014-254/how_to.html#use-ssh).

## Debug Qualcomm TEE

Qualcomm TEE kernel logs, also known as the TrustZone diag log, can be used to debug
                errors that occur in Qualcomm TEE.

The TrustZone diag log is available in the Linux kernel driver, which redirects the
                logs.

1. Connect to the device as the root via SSH.
2. Capture the TrustZone logs using the following
                    command:

        cat /proc/tzdbg/log > tzbsp_log.txtCopy to clipboard

The error codes in `tzbsp_log.txt` are encoded in hexadecimal. You can
                run the following tool to decode tzbsp\_log.txtfrom hexadecimal
                to string.
1. Navigate to &lt;TZ.XF.X.X
                            path&gt;/trustzone\_images/ssg/bsp/tz/build/tz/A53\_64/&lt;BuildFlavor&gt;
2. Run the following commands using python
                            3.

        python3 print_tz_log.py -l tzbsp_log.txt -e errorCodesDict.txt -t <TZ.XF.X.X path> -o tzbsp_log_decode.txtCopy to clipboard

    For
                            example:

        Python3 print_tz_log.py -l tzbsp_log.txt -e errorCodesDict.txt -t //crmhyd/nsid-hyd-05/TZ.XF.5.0-07927-KODIAKAAAAANAAZT-1 -o tzbsp_log_decode.txtCopy to clipboard

For device log collection, the TrustZone diag log buffer is part of the RAM dump,
                which can be parsed using `qsee.elf` from TZ.XF software in the crash
                dump parser tool. For offline or off-device log collection, the TrustZone diag log
                buffer is part of the RAM dump, which can be parsed using `qsee.elf`
                (trustzone\_images/ssg/bsp/qsee/build/${tz\_bid:EACAANAA}) from the
                TZ.XF software in the crash dump parser tool.

**Debug using secure crash dump**

You can debug QualcommTEE using the RAM dump. The execution region dump of Qualcomm
                TEE is collected via secure crash dumps.

Devices that trigger the fuse with stage 2 sec.elf are known as secure boot-enabled
                devices. To debug on these devices, see [Sectools v2: Secure Debug User Guide](https://docs.qualcomm.com/bundle/80-NM248-23/resource/80-NM248-23_REV_AA_Sectools_v2__Secure_Debug_User_Guide.pdf).

## Debug TA and CA

The trusted application logs, also known as Qualcomm TEE logs, are used to debug the
                errors in trusted applications. To debug errors in the client application, the
                kernel and journalctl logs are used.

For online or on-device log collection, Linux collects the Qualcomm TEE/kernel logs
                at runtime. You can connect to the device via SSH and use the following commands:
- To collect the Qualcomm TEE logs from Linux:

        cat /proc/tzdbg/qsee_log > qsee_log.txtCopy to clipboard
- For client applications, to collect the kernel and logcat
                            logs:

        cat /dev/kmsg > kernel_log.txt
        journalctl > journalctl.txtCopy to clipboard
- For offline or off-device log collection, the Qualcomm TEE log is
                            available in RAM dumps along with the kernel and journalctl logs.

## Debug on secure devices

As part of the secure boot procedure, blowing debug disable fuses disable debugging
                capabilities on the devices. This includes RAM dumps, INV, and NINV debug on the
                subsystems.

The debug policy feature allows control over the debug capability for a device
                enabled with secure boot.

The debug policy image enables debug capabilities such as JTAG re-enable (INV debug),
                RAM dump, and TrustZone logging (NINV debug) on commercial secure devices.

For security reasons, the serial number of the device controls the debug policy for
                secure RAM dumps, Qualcomm TEE logs, and JTAG.

Enabling JTAG on the Qualcomm TEE subsystem disables the device security with respect
                to hardware key generation. As a result, existing secure storage like user data,
                SFS, and RPMB becomes inaccessible. Sometimes, the device may prompt for a factory
                data reset. Use the following command to debug on secure
                devices:

    <meta>/common/sectoolsv2/ext/<platform>/sectools.exe secure-debug --security-profile <meta>/common/sectoolsv2/<chipset>_security_profile.xml --generate --outfile apdp_out.mbn --all-flags --sign --signing-mode LOCAL --oem-id=0x1 --root-certificate=./RSA-OEM-KEYS/qpsa_rootca.cer  --ca-certificate=./RSA-OEM-KEYS/qpsa_attestca.cer --ca-key=./RSA-OEM-KEYS/qpsa_attestca.key --oem-product-id=0xabcd --serial-number=0xabcdabcdCopy to clipboard

Ensure that you configure the OEM\_ID, PRODUCT\_ID, serial number and keys, and
                certification paths appropriately.

For more information, see [Sectools v2: Secure Debug User Guide](https://docs.qualcomm.com/bundle/80-NM248-23/resource/80-NM248-23_REV_AA_Sectools_v2__Secure_Debug_User_Guide.pdf).

**Flash APDP on device**

To flash APDP on the device, run the following command:

    Fastboot flash apdp <path to apdp.mbn>Copy to clipboard

| Stage | Full dump | Full dump | Full dump | Mini dump |
| --- | --- | --- | --- | --- |
| Stage | Applications (DCC and scan dump)<br><br><br>                                <br>aDSP/Video/RPM/ SLPI | Modem/Qualcomm TEE/Secure dump | TZDiag | – |
| Non‑secure | No debug policy needed | No debug policy needed | No debug policy needed | No debug policy needed |
| Stages 1 secure | No APDP image needed | No APDP image needed | No APDP image needed | No APDP image needed |
| Stages 2 secure | `--nonsecure-crash-dumps` | --offline-crash dumps with device serial number | --logs with device serial number<br><br><br>                                <br>Or<br><br><br>                                <br>Encrypted TZDiag with<br><br><br>                                <br>`--nonsecure-crash-dumps`<br><br><br><br>                                <br>+ TZDiag encryption public key/exp in devcfg can be configured in<br>                                    the following location:<br><br><br>                                <br>/trustzone\_images/ssg/securemsm/<br><br><br>                                <br>trustzone/qsee/mink/oem/config&lt;chipset&gt;<br><br><br>                                <br>/oem\_config.xml | <ul class="ul" id="debug__ul_lyz_lty_21c"><br>                                    <li class="li">Apps minidump:<br>                                                <p class="p"><code class="ph codeph">--apps-encrypted-mini-dumps</code></p><br></li><br><br>                                    <li class="li">Modem and WLAN:<ul class="ul" id="debug__ul_ocm_nty_21c"><br>                                            <li class="li"><code class="ph codeph">--mpss-encrypted-mini-dumps</code><br>                                            </li><br><br>                                            <li class="li"><code class="ph codeph">--wlan-encrypted-mini-dumps</code><br>                                            </li><br><br>                                        </ul><br></li><br><br>                                    <li class="li">aDSP minidump:<br>                                        <code class="ph codeph">--adsp-encrypted-mini-dumps</code></li><br><br>                                    <li class="li">cDSP minidump:<br>                                        <code class="ph codeph">--cdsp-encrypted-mini-dumps</code></li><br><br>                                </ul><br><br>                                <br>See [KBA-191202045020-1 (ZIP)](https://docs.qualcomm.com/bundle/KBA-191202045020-1/resource/KBA-191202045020-1.zip). For more information, see<br>                                        [MiniDump Software User<br>                                        Guide](https://docs.qualcomm.com/bundle/80-P8754-71/resource/80-P8754-71_REV_H_MiniDump_Software_User_Guide.pdf). |

## Qualcomm TEE/TrustZone diag log collection on secure device

On the secure device, the Qualcomm TEE/TrustZone log that is collected from Linux is
                disabled by default. Qualcomm provides an encrypted log feature for logging. Follow
                these steps for enabling this feature:

1. Generate an RSA key for encryption
                    using:

        openssl genrsa -out rsa_key 2048 Copy to clipboard
2. Show RSA key information and modulus
                        using:

        openssl rsa -in rsa_key -text 
        openssl rsa -in rsa_key -modulus 
        Private-key: (2048 bit) 
        modulus: 00:a0:48:99:99:83:26:65:57:fc:75:52:25:45:53: 
        92:fc:27:29:cb:14:35:94:7c:89:bc:d4:0a:c6:3d: 
        0d:6d:8a:7d:72:1d:e3:4f:f0:32:66:41:a9:f6:c1: 
        2f:79:aa:58:ea:57:3b:29:6d:cf:40:33:4e:ad:ec: 
        bf:78:44:4b:28:52:c8:e3:6e:77:01:e5:a3:c6:25: 
        65:8c:8b:cc:32:20:2d:29:58:03:f0:d5:b7:f4:c0: 
        d6:09:b2:8e:59:c1:3c:ac:e5:61:04:36:78:e3:da: 
        95:b3:e3:b7:71:90:50:ee:a9:70:5a:15:1a:af:d9: 
        a5:4f:c2:70:f1:f8:f1:67:d1:78:0e:b8:95:6e:93: 
        73:6a:23:f1:31:e1:e2:49:ff:18:54:a3:73:d0:70: 
        91:de:7a:92:53:11:aa:cb:b0:f9:d0:e1:83:9f:74: 
        67:bc:1a:89:6d:b1:d2:de:4f:ab:3c:1c:63:c9:bc: 
        75:f0:c0:80:fc:db:73:d1:8a:e3:f4:60:57:dd:66: 
        f1:3a:fa:18:ed:7f:47:72:3e:49:50:94:8e:19:ae: 
        6b:69:62:3d:74:ca:44:fb:d4:1c:1d:59:43:30:31: 
        0d:fb:ab:70:44:9d:d9:d0:ce:cb:43:f3:2a:98:a4: 
        83:e7:76:ae:a8:b8:ea:63:64:e1:11:1b:99:92:b3: 9b:3f 
        publicExponent: 65537 (0x10001)
        Copy to clipboard

Note: The modulus is used in the
                            `pub_mod in oem_config.xml` file. The
                            `pub_exp` exponent is usually 65537. 0x10001 is known as
                        the `publicExponent`.
3. Set the RSA public key (exponent and modulus) in the
                        `trustzone_images/ssg/securemsm/trustzone/qsee/mink/oem/config/<chipset>/
                        oem_config.xml` file. 
    Enable this feature by adding the following
                        lines to the `oem_config.xml` file
                        using:

        <driver name="NULL">
        <global_def>
        <var_seq name="pub_mod" type=DALPROP_DATA_TYPE_STRING> 
        a048999983266557fc755225455392fc2729cb1435947c89bcd40ac63d0d6d 
        8a7d721de34ff0326641a9f6c12f79aa58ea573b296dcf40334eadecbf7844 
        4b2852c8e36e7701e5a3c625658c8bcc32202d295803f0d5b7f4c0d609b28e 
        59c13cace561043678e3da95b3e3b7719050eea9705a151aafd9a54fc270f1 
        f8f167d1780eb8956e93736a23f131e1e249ff1854a373d07091de7a925311 
        aacbb0f9d0e1839f7467bc1a896db1d2de4fab3c1c63c9bc75f0c080fcdb73 
        d18ae3f46057dd66f13afa18ed7f47723e4950948e19ae6b69623d74ca44fb 
        d41c1d594330310dfbab70449dd9d0cecb43f32a98a483e776aea8b8ea6364e1111b9992b39b3f
        </var_seq>
        <var_seq name="pub_exp" type=DALPROP_DATA_TYPE_STRING>
        000000000000000000000000000000000000000000000000000000000000000 
        000000000000000000000000000000000000000000000000000000000000000 
        000000000000000000000000000000000000000000000000000000000000000 
        000000000000000000000000000000000000000000000000000000000000010001
        </var_seq>
        </global_def>
        Copy to clipboard

Note: When the public key in the
                            `oem_config.xml` file is updated, ensure that there are
                        no new line characters, tabs, or spaces inserted in between due to the
                        Notepad or Wordpad editors.
4. Enable the encryption feature configuration flag from the
                        `trustzone_images/ssg/securemsm/trustzone/qsee/mink/oem/config/<chipset>/oem_config.xml`
                    file,
                    using:

        < props name="OEM_log_encr_enable" type=DALPROP_ATTR_TYPE_UINT32>
        1
        </props>
        Copy to clipboard
5. To build the TrustZone devcfg image, enter the OEM ID field value and sign the
                        `devcfg.mbn` image.
6. Flash the signed `devcfg.mbn` image
                    using:

        fastboot flash devcfg devcfg.mbnCopy to clipboard
7. Collect the Qualcomm TEE/TrustZone log
                    using:

        cat /proc/tzdbg/qsee_log > qsee_log.txt
        cat /proc/tzdbg/log > tz_log.txt
        Copy to clipboard

## Qualcomm TEE/TrustZone diag log decryption steps

1. Download the Python decryption tool
                        `decrypt_tzdiag_qsee_log_tools.py` from [KBA-200917004544-1 (ZIP)](https://docs.qualcomm.com/bundle/KBA-200917004544-1/resource/KBA-200917004544-1.zip).
2. To install, run the following
                        commands:

        Python Version 3.x
        pip install pycryptodome
        pip install cryptography
        Copy to clipboard
3. To decrypt, run the following
                    command:

        Python decrypt_tzdiag_qsee_log_tools.py -pk <RSA private key file> -a RSA -I <input encrypted qsee/tz diag log collected from device> -o <decrypted qsee/tzdiag log filename>Copy to clipboard
4. After successful decryption:
    1. Navigate the plain text of the Qualcomm TEE log to a readable string
                            format.
    2. Convert the hexadecimal encoded error codes to string,
                            using:

            print_tz_log.pyCopy to clipboard

Last Published: Oct 14, 2024

[Previous Topic
Customize](https://docs.qualcomm.com/bundle/publicresource/80-70015-11/topics/customize.md) [Next Topic
Examples](https://docs.qualcomm.com/bundle/publicresource/80-70015-11/topics/examples-lru.md)