# Features The Qualcomm Linux kernel offers the following key features and advancements related to the mainline Linux kernel on Qualcomm platforms: - The Qualcomm Linux BSP is tailored to support devices using Qualcomm platforms. - The device tree corresponds to Qualcomm development kits. - Multiple customized build configurations are available to suit your requirements. - The Qualcomm Linux kernel is integrated into the Yocto build system. - The Qualcomm Linux kernel is aligned with the upstream LTS kernel. Qualcomm-specific additions are maintained separately. - Support to configure, customize, and build kernel images that can be flashed and booted on devices using Qualcomm hardware SoCs. - Virtualization support for untrusted virtual machines using the Gunyah™ Hypervisor Software. - Periodic LTS merges from the corresponding LTS kernel branch to get the latest security and stability fixes. ## Boot Qualcomm Linux supports systemd-boot as the UEFI boot manager to load and boot the Linux kernel. The Qualcomm Linux kernel, in this case, is built as an EFI stub. ### Boot flow and architecture A cold boot refers to the process of starting the system from a power-off state. The cold boot process involves the following steps: 1. Cold boot begins the execution from the primary boot loader (PBL) that sets up the initial system for the eXtensible boot loader (XBL). 2. The XBL performs wider system initialization including DDR initialization and loads a UEFI image. 3. UEFI provides a rich firmware interface that loads system-boot as a boot manager on Qualcomm Linux to manage the OS images. For more information on complete cold boot flow, see [Cold boot architecture](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-4/architecture.html#cold-boot-architecture) in the [Qualcomm Linux Boot Guide](bundle/publicresource/topics/80-70017-4). ### Systemd-boot Systemd-boot is a UEFI boot manager that executes EFI images, provides boot entries, and supports unified kernel images. Systemd-boot supports the following components: - Boot entries: The type 1 boot loader specification entries are in the `loader/entries/` directory on the ESP. These files describe Qualcomm Linux kernel images with the associated initrd images, and other EFI executables. - Unified kernel images (UKI): The type 2 boot loader specification EFI unified kernel images are executable EFI binaries located in the `/EFI/Linux/` directory on the ESP. For more information on the boot entries and unified kernel images, see [BootLoader Specification](https://uapi-group.org/specifications/specs/boot_loader_specification). Systemd-boot is a part of the systemd package of the meta Yocto layer. The Yocto recipe uses the `systemd-boot_254.4.bb` recipe file to build systemd-boot, and the `uki.bbclass` recipe file to handle the ESP image generation. For more information on systemd-boot, see [systemd-boot](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-27/platform_software_features.html#sub$systemd_boot). ### Qualcomm Linux kernel as EFI stub The EFI boot stub allows booting the Qualcomm Linux kernel directly without a conventional EFI boot loader. The boot firmware can load the EFI image as an executable file when the Qualcomm Linux kernel is compiled with the `CONFIG_EFI_STUB` kernel configuration option. In this case, the firmware loader navigates to the `EFI boot stub` location in the EFI image `drivers/firmware/efi/libstub/` to boot the kernel. For Arm^®^ (Arm64), where compressed kernel support is not available, the kernel image functions as a portable executable (PE) file format or common object file format (COFF) image, and the EFI stub is linked into the kernel. For more information on booting the Linux kernel as an EFI image, see [The EFI Boot Stub](https://docs.kernel.org/admin-guide/efi-stub.html). ### Boot images and ESP or boot partition The ESP or boot partition serves as a storage location for the `efi.bin` image that packages systemd-boot and UKI. The UKI includes kernel image, initramfs, and kernel command line arguments. The UEFI firmware launches the UEFI boot loader and loads the kernel boot images. The ESP is formatted with the file allocation table (FAT) file system supported in the UEFI specification. For more information on the UKI image format and the corresponding support in base Qualcomm Linux meta layers, see [systemd-boot](https://docs.qualcomm.com/bundle/publicresource/topics/80-70015-27/platform_software_features.html#systemd-boot). ### DTB selection All the device tree blobs are packaged as part of `dtb.bin`. The UEFI selects and loads the right DTB for the Qualcomm Linux kernel. To understand the boot time DTB selection, see [Platform support](https://docs.qualcomm.com/doc/80-70017-3/topic/customize.html#platform-support-0). ## Platform support Qualcomm Linux uses the following procedures to choose device tree files, change kernel configuration, and build out-of-tree kernel modules. ### Platform device tree Qualcomm Linux supports a device tree overlay feature to maintain and merge out-of-device tree blobs with the baseline device tree blobs. SoC device tree support is present in the kernel source in the `arch/arm64/boot/dts/qcom` directory. Qualcomm device tree source lists the device tree and maintains a clear separation to contain downstream additions for provisioning the upstream aligned base distinctly. The downstream content is hosted outside the kernel source. The out-of-tree drivers maintain their device tree overlay additions outside the kernel. During the build process, the corresponding device tree overlay is merged with the base board DTB and a unified DTB is generated, which is used with boot images. The following figure shows the build process for device tree overlay, and DTB generation for custom BSP on QCS6490: Note - Base BSP follows the same build process without the **addons** downstream device-tree files. - All supported Qualcomm hardware SoCs follow the same build process for DT overlay and DTB generation. ![../_images/device-tree.png](data:image/png;base64,UklGRsRWAABXRUJQVlA4TLdWAAAvqMTbAOZh0LaRpJg/7Nm9ZxBExAS01IVnMMy+ldmhsw/+dR3mAZ6V7o96t5EUKjl+4FjSRBOeLDf/0wYFLVwCbdzg5K+pndjR95ZcjbvjJ2+3LDRP7l9Ut/+Ra/jIiDoVVgmAXUuSM/INGw4spoEDFw4UHLhwYcOFw7Rw4MKF8hIcuLFI0HvL1iIxMe/uO/e8e+/Jk5knz6muWP4+or9gfqBAyaGOxxrLu58QWpMyRZ5sx73yuh9xSt5LRdc10kbH0PoKmUeKPFKheHKF1vzChGBTuUb1BJ8cfFBUcKm8f7LI2+Ly0hqtofLccxWDQlt7oyj0jx458pORI+lEIiORI0dG4shBIZFIZIdIJBJZtyh29Lp1VbjA9j9udi8YDP4xGAzmKRWLwWIwWFsxWCwWi7sXLM7uI6MVdx92Qv9pQZIkppVy3mltwvjELAYZS6sdPvU82/5ejuTUR9iPMCHDDjeccEKGE3Y4IcMOJ+yQoUJ+hA4n7HDNl+hQ36Kxv/d5nvd3/nUOy7CAjQfbWacl30BnArwlUAHlfWMoUyh5d2QmIlQZIwY/eW8OptI1nTW6Ix6ZWTQGJyh5Kij5xs4B5U0DHRWaSbe81lTL/Nc0UFp7GMkMCGYzmcy6kmlgDScSfgK3gAYGBRCQFwGWHNHhEDgb9roj781sJhMVZThKGxywcEBGsoSYDLpQJTcwtG0OAw0WBwd/DP4YDA4uBgeDwWKepWJwsUcoBnuMYnFvUZja9gaW/ks4uDhYDA4Wi8XiYC4hGCwGi8XiweBicWyLg4tzB+2/LMi2wra5xEqtSvF5onrmYaCBz/f8/7/T+/7tHps0Kt7hdfeu3+93+Xq91WX+qPcyjYoffufP8/t5VPJMo+KtLm7gcck1hYqSxz2v6VNYPPZ5TZ1iy39b/tvyX9Iss6db4qG8yOzpllgoL4p7MXu6hXdE6b93upTF3Zg33fKo9+Mis44k/XenCW4RlbtdUtyPGdMtb3SanwM6L3HxLw5SfaCRvPHQaaL6Bstu0S9b/ttcIlMW7mQqQ1rlBWWRXckLKiPLo9orR1ksckNqdgVx3eKxHMIBTV5s94TF4HCAZlTZAbWnMe7xtniI81ZehHBk16o8GChh0bvzYBspjbFBAGLQyEM4FlkSDeaj1/CNBzR4cb+EZwNwJyVJcbvkrt0RiK8mL86DiKz/fI2GJSkuULt2lzIfR17cJpn/nv7VQRtwvU5jSFBA8CziAgMXqD8uaAOw126OyAkYPoi0QMKzAjbk2QcFa2yRqoNpdy9Ucs/4bEiwZw+ROnVMnd28MMl9h/A7ZEjH7BnBSL2JFULIfJ36rty3ASmlbcK76PXO7l2I5L5T35HTDvN6B20ZkLQreb3Tqb80wewZsagj38QKIWS9Tv1Kull7vV6/OCBhJzwYZHEXnSeDMALoSKrdkKDnPj4TOhEgnfqurAMGofRS0RGP6B+F/cO7UOmVLd2fGK9T35l5QBHy3e5m+5eCcVedEsVMREM4RLH7EfDcd+Izod7xQVD1DmibhAENE/URkewxWee3/tjyQQi4mQFu3059R9Yzdi+nlCaDv9hhupXS6pFzKYHjWEzJSjonBizcjJTOnyG2zm5HkHJ/xa6yd+3rRx7dAT5nNB3Gqp/YMq7JwLnogARgjF1NFxYvjIxUv7Ixod7qYssH7Ad3cb0DBHYrX6jXFzcmjpzbmKgvmjNcPH8GSOpHzq3WT2xdvl5ntnpndyMouac39RNbF5xrb15vEogYK8i4NiY4VwQIb0DGMAISPbyCwK6kW5+65ZQP+O/E1ioaYoXqt03ULfyy+71Ox00iWf3K+TMGINh2P4KT+9e/Ga7yxrv1tQcAKPAx1sSJLUQSucIgevcFKyOOgEQPpyCg3u8YJyRgWftKSkGTcK+D4H2OfMFkndmGLoHvt/NuNtbXHmksBwCLMgHe3CFXJIgn87cjH+1gSqaOXxDY8ssHIfFJ3QrX53cmPucrYUIR225IYHJPOrHl6tEJFGMJca5Ey+6skY7Sxk50aL7iFQTqdJwsH4TDl1zhElyENibqkE5sCZzdkeDknjcRit6IXIkTHL060hHUZl4I4rc6R6p8wENwdgm4YlX2V5IjuuE7LGLbDQlO7sVukt59Ue5mYa5UpCOPblOMhE5Lq3VVEHCkcAwBsKPp69+8MUHuMK16wqrMi+gTBNvuR3ByL05ztTHhnCZuIJErUsGGDsuCmhFQ+y3xKXGn44twQsC+jvm7lPDfSUobExe9UgzipPNn7F+IYpuod3YzXXfg5J7bb/l6w7mUJkX7LXkXUa4oEPKIQie2RkJ6fThGvNV5EjiGUDgYuqMbgcNxGkdDGMIQ1RmaETr62oOn9LA0lw/KhAhKGCOerjtHzp0/w291WrJ8wH9udzkBAcdpnH0e2xA2vIy6yHwcVp45WJorApQIJei+kYq8d7GItzqE0jjiOWAfaLCze0Q0vEyH3+oqKR8I6Mii8C4+RmoI5E6nM4ZqrPYP9F/pL0GaRYx8kaR5WQntLGLOA/8SSXErwLUNwC+RF+cEErF5WUnx3EE/95eCNkB6q7NrCYuH+8KdeUtjPF8g5w5aKVvgT8RfGlUM96BIhftCwOa3Dk4b7hfLV8UQEJ0DaimKDVLXblOMvo/PS2yQrRpsQFp1FMXctb4jL1qDrFa1Zf6o0BMQLIgLwZWpCRgyaSG6Gnzx9bCopSsC0Yco0rgomMNF0oMAtyuG5LWpfqm0jtXWptGVl10N/jn7VGGto4DMhSjSuMMxMKeQ3huAu2JIXpvql0rrWGltzJWXcxUz3vHOmwZ21Xfm5SeSpQE5bx+vwfLnCgxAHzSYF9j1Xkj2bRWEYBO0Lkk8JMC8fWbmjncM1HNwU3wBuCsW3a/fMqz8VLdoINVNdYu1X0rdEgu88j5fLmNm5uNuEr3jfmdmZp6E25QDLQLAn2Q4SgpCsMXSVgJJTiUQFTd5kT0BuSuMLa4mUp0b6hZ7v5S6JQI4DLchyhBeyTkTbswZFUZRUEghGAuQQIqNaSNIKoBEi63U5A+QXSHVSKoXjXaLweiWaOCdA3Ldcb4oUbRKKZlXgKALgsJXjisFtidT+nc/ESOjUwJi5kmAnLn91QGeu4YOEEjLJF/0rl11vMuwTylIF+kNKylZaembjG0VPwDuCmP9bKmu0y+Ndktn9kupW6KAj3u5jJlTHF8ZwjuT3eoWc/1uiLgChEF93Jt+ZkDCYouAAK43DBDf0m7mtwMxkwQRloCMQIKSyikrHnoDaFcI2VJdp1+a7RZzS3fQLTHAYYqpsMYaCg5/4Ht/S3RaVn0cbti3GxkAGB54AwVh0FDwsRv7lBU6BAnTKsgIpFBKIiu8AbQrWLa1qdUvjzrljXaLsRVpdEs08HZRbHlnerkMvGHhPABJeJ9CFMYfrSkI44CbOdC6JEZryAYZi4QCf+QNoF2hVE51rX751Ca7xWR0SxRg9z3r1CkLf0Bwbz6Z0ptuP3WF9KTWk0IJ09vf8CFf4JEIxUDWssB1Rbyq9oupiW6JF77oXVJPkqjwYZFPARtIQS4JKwqyhjfgusJVvVRbNNItEcO35GIL/CRREWR37sw7EwmdxJOwW0GQizMsokcSKR9SqRZ3A+2KGNXqlz+u3y3DGd0SHa6G9z5/tEYllvQtw31KjoBPYwjxedm/+4I3rADJ22cEiVAMpNK3rLXTgHZFlOr0S7Pd0hn9MuiWSABiHtlPRTRWmnnSQD6OixgVtd8K4B+gV1ZdweZITCKkICOQPq7m2m8B7Yoo1emXhrul+7a4sP8W3aURTdrhBSJwZsAaBUEAVsbx+E4x1SkFGYHEVUGDFPkBXldIhCGprtMvDXfLwtYt8eAYQz0Y82EEsBdwcBSChEAArMBx+A4z1WEF6SARQI31MYTcFXlIquv0S8PdkkvdEg1gLzZ/2AV6AYkjECQEAVBNs+b7tkylIGOQAnGNjdoAuivsqa7TL013y9jWLWKuYsiZyu6LQtkHldgXMLuiUENcrrB4a27o3FMVERbJXv3EIIDsiiIN8WCFF5sKTyqRUp2mS45OfCphxOhU8iJGjibnIZBqr13YT3dIVI0zbRMZaUjdnqCtW4TVmLSJjLgypnOeJ/KI8G5pBun9suW/Lf9tXSyLQpSxpVtKCO8Xz+WFLISJMd1SQniNLCAMKrowhIEzYSC96igABvXRv9rAb2UhVG8NSMuauKl62FIdDxqGb3icFg9wXjvVdCJGvvuydMLOjUI6dXE6dbV0av+thOrQnYtWEa2iu6WJNL6NAYQ/nzpA8FdK4xtDQfjzaW5cpvV7geD0EMNbBArfuKMhivtl8wh/Pl3khhC9gbHIvntF0dCwjWerG/XfYSH6pztkW5G7RnFLaBSPb57TB0h+J8OgDYVmpsRF78BUCQ0EXFD6NbQN2pBmr9OxARsg3RkGb+QcWfEQY5y5xxZHJMY/YeF1OuNgDajebOKICqeWBIbYetPJjqRwbI+82LLa53AiAmPxADQl8mmMsTP6nKOciOBzAosLICmR73M0xlgZfc7RW0pKmG4xP7qr9G1ASmn1aGjMtAgcnwNQGzvhRz9HUgaMfxGQCGNX0QFDYBizIoosBcBRS/9k60J4bEyEcy8cPbpoXjB8oTvEL+AEgVP8HIAe47LhyJLiSfRzi+fPHN1ZeEDx5UeB4AM85YUAcNFc5ioAgICTxFHq2BZ9TnCEEFBiYBmcH6CsgrPDohpLuUyrJwPOjX+1cQs4YF++jvBjWp58Dng9CyghsPwnW1AXdjH4QPghgCCe/MlHJ4L+RRAT/YsEGwE/lkXk76A6jEtviEBU4UCuu8efgeehYw2HggT82Bb9k63g3IL+5Lkvx9IbuUB6wFlQU7aqRY6VdiM8a8zKbmj9wPEkkly8fP3o54g/qyox8cS0qEXww1xlS4qBSRipepyNWZCzxgDN78QXbnNhmcTeq13cmKANPuda+lU6BV2h8JQRG4Yf1iqXlrJgyEzCSFXgIJRJwQiNAZnfiS/c5lxYJjF2Aa0ZgmOj3Vxyg4e5TNyAQ+QfJGkWccPww1nl8rYhA0BK26WVGRi2B24SeZ++ylzu6JTSgNBtvsqcy52jIECVeILlYMHyTi7hiN7IpbeElWIdAAOX+Cp+QE4UlkmM/RKqnofPFK0FhKqlXz0KiXerhqZHj+KxdNqqgB/uKWMigKY38tvy/Hi5uWIOwvxBUNlf75VT060Sb1N0EDr+YiMw+TTSNmk4yO5zshf8Al7iHRMqzoHorMjNCFD0fJRBFWUSexf/IqW0McHttzimRAxEOnpR9OEBDvCj3H5rIlBdvk7wwzzlElNGgFzq9D7Db3LsgFvRZUDK+8Qq7XY1ba4Ayfw4IvVvgET8xS/A+JE0TqwGTCQGBQ9a4pXQV1WYA1llhQ/ik4CMBP5QcSYxeo94DEKdXtF+Z2ggE2ABcJjYIh4jqJ0uA6BeaQDpRjk1vZNtJaU5i7Ra5dSlnrizu/PbHfPjWHSx86amW+w6GD/dMI+RDNjDR6q5stwcIO3wkpaOE1uWhjfWL0ynND8+WE/9WvCfAEw8FbHsujK+scG19uwIhtgFnlNYmyuG1QLDPBIjnb+tSzAcDq84kzh8jKF8AWecGX94GSaTw9BIhF86bFMu+0vHw7263iu76z0rpkxNG0NaOm73/vLS+0Rbgds5m6voK6XW1CUAQGWmtpINMDj4RDIXyObHS8GWpeFQOKFQTr8WOoCJxyIWwoocWAbfdcPNganp1DIkxgoikzlzBAluNlfSHJTVKs0k8R6wL8NdBu4NQ6nSblFxXAvu6+eeuIGxVxBzphY5CvBzuURqKMoAsIcfqNCXBADJlpXhfZC5G2P1oYOYeLjyKdCVmAMYrGKJz7EDAKxQ+D7dGWtzhYhE5kiSVsmOt6JMEnJ0l01dwggq4z0ojrPyFrJ158fLvqrDlhK0VKFssVTLwQdG8pStUrFlZfgyanP1vc8+8gDcxEMOaHw4j+1gtYD2AmNRca0FFpSSxMSZCaXGyEwSdF0W3oOuiGoa4itqFdBPaenVD/dUxGUJqKnuZum2kGlfCRNIsBGolJV/vLgYZOLVRuIDjSfDQhGWEmcOy9tUkklyvljvVVSQEcWcLt+9ZQYvMIBy3BwA3LFQ9exSx2240MFIMRsy6t//9XegL6AlfscPpZTmhCOL2MSowk1Zlix3dwWZJOU2V7yqIU+BShXXMOIqy+y0CsCbmqgRcTm1ovAzSrJl5Ui6/Em/uXkDHcjEZ3FahYuHmgZatdsNMH1NidMqyCQZ18zio0EDsOpq737uzo/brY6+rE+NLq3QAcdxzVDOkhrLHi7+eg8KHSBmY19j/AkcFBkEfwEy8WADuLfy6luuNtx6zzlNSGSOFOUNXqCyjM0kUVb3opTSDopqvuQJWxlBwIMqN1e450oXAMg3EDXC9P3yC1AtHcdik2ALboT5G8GCfpof/3Ed3Bjw+2/BTHyT/rBcDt1/q4KHplNqCSylLDMHcSAnRMZaLkVmkpRrZlHzrNpXe8eVdI8SCJNopGZE+28fXxlFiJJfQNVMi3h4iY+wo5zeCjfBeKSMxuIMkKB+xuZKMmko6ZpFjMfHGIJU/7+mOk52B2QQInGQYnrUufjSKI9NYUgc8MPLQE28nwN71ZIPisZqxhmetTmVZBJSQjY81WyShtAB+3xOvKEqDcTJBuCXgja8HV4GopahdGRRjxJf/QYOjXrFbEPDNj7S13GwIGoZSuVR4ivfwNqRcM3GWJbGahew+a3LEfmqGEKQiFcNNpJEuqrauiiVlOrilOpqm1L336pzj5+Eb8yL+6UI6zV8o92M67uPgjYKIlTVFh6WARsDMlQnIB72gjUMUlR56eZV+61/8ZEpPTPsmB38qaqf1cHLpDrr5WF8cEqfMOxYb4Wpqrfvt4JAqrNi/gVqmLGCHQZP38EOSTXO3gVrWLEmMCVevicwzFXpPJN7dc4zaRiSJOc8VAuP8W5ItXQlGuP1jTf5ed9grwpn7t5bVjtzt+IKJPo/xUs8gJtOeKW/FHnaAeR0wo2VLk/1prTSKcgC5nTCB6trmX7XkXELUhAApxNeb1LkaQeA0wmvPM/vBATA6YRXnhR5AgLgdMKrTYo8DQFvOuHLqpMiTzEWwOdTPlB5UuQJCHDTCa8+KfIU5EvATSe8xjy/ExDQphNeZ57fCQho0wk31sY+KfKUZI94YNMJN9amzqTI0xJ7ewag6UtPxsEyfztMohi/HSZS9vb2tvzHXoI1ixj0ZZvMBvbzCVIsZAN52X6G5swEeWFRdAW5HqTVw6KMlLjLxP/03S/jRaAuL7quZLAGGRVe3/JFiaddIZBTckTjp+o15tmsM3VPGFyBAfJi+2GzdLszdN3Jayzy4l88pLeHbHIhUqpfNnm+HnNBNzov8XRyhxaqD3Hx5KauiTWbUBNnukHgx7SxUf/kUEtxDmifDdZajhHOnzh9+YGcWGnLw4n449GUHTPi4lHZGmdOV52aMLCwtBF0mp19RljbW0qNMzFRwsFikRBpyi063wiDZo00NpSPZsHA8tswacyWFda2Ni7aLbXo7BGPQLOOHPx8DeHQbD355bxBPndFqRl5+bqKpMp+/lXmR9n32vm4ozm+d8Vo0Ho5yVmRFE/lZ3zkxGwtNSBpjO9dsdaRr9PGlkhDdfeJmXGP5U3FEJtT3SrJ7eRkxf0O3U72YhLjTF5d0OqIrPjEMraU3yawyBqG1oYpBU0y05BDdxiLqoH9rXhSIi4z5SEHQ+eESV50XVfSQRoelrSDpGOY+Nt32pfGwTqYnoDpjEzbsDPOQKrTqeLfCiDnPLTIr/4KncWh7rQ7etaqjgD9R9xAfuBF/EpcTCKQKvg9S8QvXDKkyLP+Chy7ah91EcptOenpisHGriOw6nSKo102IAEwlMi3/mIKbIb2BtZOla6YulO/WKdm4fN08iIq+o3jGUCY5AH4wZlHF0mD0Vq1Ckp9U170FwK40ayZq36xTrn4eTp5ERVIAGhbnF+o0gF26eJo2IIqFZT6pvzoL//xobGNugjJhKZwqn6xThYxFTlPJyDv4y8er37TXAbg31G33Fz5RyuBTRiF5ReylMH1PKHAMGJ+uJGWjrNx1SlA9U350F/eY0azhoao6zhVv1gnnmpTYfN0akLRBKCFCQLAIC38CkLD2CgAZksFuDNPKCKZC2Tz46Vgy1UpYPVNge8v3zGiKQ9jDQtJ/wtABiUMeBMhEFNtKmqeToF88+FeaUzE1V/vCQCMsZbBKDkDjBIdBFkqwL15QiFVAA3Miq1KBay+KfD9xSJeDA+7a7YAdsIcJtMTkBiPQM7UUsRUmwqbp1PG+pyUWmYC4hAA3fHo7VqlTeICjQJccDQOOIDwcWCcmDWGYEPkXG0CVt8U8P7aM/5nlEFzp81BM4Kx5ZXtymtByM6lrtUt1qmZeapNxczTCauwu4HmtdOBlNX1tLkixEZBIksZ3FfCBFJsGWuzq0w+1jc1CBf9Zck6XTDdaXNX11B2wrSxrAQhO5dCW6z3ZLEESxhQ9ihknk5/J6U0x+6hoBJJSUZB/bxbEIrclIirkKhqP1fXxsP6poy0u+mvctapV//ehewoiiXWdWiJmKgHALWLdeKpNhU0TydLiLYMESOZQACyjkcq8w2PuxW4J7Q7owRbrW5qd9WQfIHvr9JCxtUv1glKUOG4QubpBEhwf2+upLlSzNjJALxPzVhGLYh5elAM7ok+iCMnwWxVuhtYfVM+9Ndyj0E2AL+EYpwHruPFuAcY9Yt1Ag67FQuYpxPaQBAWD1XQDkmoXE6SmCwV4J6Aauk4FJykcbka229Bq2/Kg/7a45Af+jgU445A1IH89ByhamZRn+42viYOhHOINBJCEHI0QBbYbot4h1IiajOBQRpXky3i63fVsHwB7y9GeRaCURPqE56FqxlsArnd2YqYp5MAFz3hogEUdmwfQ6droXMcoUrjarOPYf2uGpIv4P3FKh+EXtSG+BSyIWs2ScOYAfsQ6Sr514SoZjgzsmidropuILQZ1gyBXKOr4gn+j9UeO/m10BVTApvPnfNwQ/aOqNJHXNWxpBiYxEFSWA5tIzeo45KE/bBFfHLEJBW7hwAUH5fU/UGwrC1isv+5fq6UEtlc7rLMwwwN6YgqfcRTnbqljmDoMImDpLAc2koNIjaPO9BWtyrsi8Mkwj3UeBCp/kpvpTeCZW3RkrGjiSGIZpOVUT73hNcpLmOr8IA+rFD2MiImjziqUz8yoEjppaL1DjBcvHy93qkvprSxE17vdOpXEEAiTiKFYKvMoF25vzmltKrYgh0B9UpAL1YonCCg8hthae4DjpKcMylneywdxwsb0+wIkz0CV5dyDnjEQjh4zcSQhRer4qHm1YX1QWKyBzNC2F9xHjFLhyNA/ZPMCQafeGLAYF/mEIPqFzcmwBWiPwM3CJikTsce588gUrw99ghci+h7ARx0YmuyTqhFCZMTBFRyI2C1oLIrXvSytLt6Dha7IbYhN7L5g/6WJSliSHZk4T+NCoeaz+rC+iAOie7mLHxfxXnEKp06RYAULDLH69+8Wkd/1wk/znRiC0guX0ekixdMgIlusw6Eq4YckOLtAW/KFpHdLEYrSKicIKCCGwEtGUdNf+30Q9MpzeHVoVW56Is+cs4Q48gmC7Zd4NvpMp3S/DiPPl4+JA9R8A62+TBd5RviynlDtpdTly5Q/iy7mZH97DHS+du6WJ9FeuOsPI+YRMWI7350om4AoXgFkWaarB95dEJ4uMXLLwUuX8cCmp0nPVxwiW/7V+n8zsTnbjY+B1GkeUBU/WJKG/9qg+xZrddPbH3qVkqTyAb86GAXge/dgfvydbK5U4xwOUFA/I1gfoKTFRjQF8qyMCqBP9Z4F5HEEONww2fjCsqCL1xXKg2nr7UMULBEUy7tn9TScbzmmytzfOW8oeZhKFF9YVVmRGQP93m2f53wz7GAPGKT4McsRkwWBG5MWEEMY8o0eeX8N9wCn9cxX3lRKU0eeRQASFsB6cg5iEaNslMHC6Q9iBSEEBSbXrSNsBs3bHO9U4iQOUFA/I2AGSivqNLtMpRg8L2iv95zhlfn2n6jm7MH84Ewx7iwbHYnkrI5MhjJanNlzrtyzpDtnD/zP9Yfd72MyB4YmEI43twsJI+YRMR/5qCgIIZ+r27lr/NnrEyGwWcQI6ZJUaRDrVrpT8WnV6zcRpAXyR6zhgDB+xndqj2YD8Q2F7XB5AQBFd0ImGmhO+RKP8uP+XExMqbBlM6AmBkeN/BViA9EQ8KjGUb6sYSXqMyZrlXO6soBER8n8meni2E5meFnDwlKxjCqcgF5xCfkk448inEkOLxAYtrg4tri5ev1i+qQCE2COxQk5lEvXw/2IInBABgAQlU9PKhmjfhAYHOBu3E5QUCs08AO3sJizAJviHEeXl2K/2zqatHo4wbjFqG7EL5MXDlfl3oowlLys8ffFJBHnLLIgjhSNFqYZKBzEGDSD7qY0vlv+LYJ9aReRwEJnkcfHi4KJUxS8Ge2W4tsLmqDyQkCKrkRsAPXkUoj+IbibCAplV14xLwZZOPrq/ePArYXR2xiFL2JyB5vdyV5xLLFxoSzsZIYlNlEEW+RPVwdkI4MaDpDsef5mxFSSqvupl7vgFt1NpCU6ovYeitiwzYXtBuXEwTk+BsBN6LOZ5w85RJnAV2vrGXBecmNPQQfXXwwxHoxuleL2O2I8qevdgth9viaEqcVkUeMcmLL282tG6zgJYprsIFPHr0fRBux2z4uRFp2otwuS8uq1+rcmEvzCZsLEi4nCKjgRkDPeg8ucdmnQcnFByFieHUWVDKq1h5dIIY8huKMHWe7Gr77b/FI7eI0X+s95zTx705kjxRkT8YibwV5xLGr0YeB+EGgVdpDySscB1Vi5grBj12kVqd1xqKKsA7LSFIguIi03mlQPGOBNxVNtQyLTlslowrb4HKCgLLMsTcCeh7C30lBNAXBMHxnlKXjYnh1ISoBr/egnZdoKdJP8+PLwHO0LO01skR4j/jstt/yxPkjsJRyRPZARpD9uYzPI6btv3UlpfQG1X4LODYm6gBEVVgg827YdUf2z2FB+y0osl1MCuD8jpTOn0F7zp8RG+AA27Cd1yK187qYLl9Hm4tqv4XJCQLwPqj4g4CVZq6sYTcEz4QgACN+LxjiioFZonyMYfauXNbHwT/baKxmRPZkfemzn0fs2yMeK+G9lusdBGJf5RAxkiQSSA49tHMgXAFIkI55iFN0i3hUThAA+PYfBPT2iG82o7rKiepECNmJRABG/KZVxBUDg/TB0O6VU8dxZUI0VkT2NJ1LH5FH3DvGkD4ndPoCIlCH4TQRI2kihSTrvOpQ30a4ApAgHfMIp+A+hqicIMDw3RvBukaAwRxzgD7fGFYM2EcaGQzYR6q2wfau0CeUVTxgX60TBDBuZFHuNDnMGFkUiuPD/JFF/XGsquyLosdqHlm0zp2Zb0Mgiwhq2DEEcsiJkcMQyJ3OoDyKGT60BodAXi6XCdVY7YkrOSkWSQAJSvltEuufXuZp3FqtBiz7J4euedkLNZgeoIT0EZUScSVtFjHiQ1jnZTWbeBsvwZHeSz81Yd0TnNj1shc+MqVPrElzifXft6NUU/qaL7VcKOR2y++t8fsWyPiS7ahEn+hPPLAh71A+xNvSvCpr5W+8Lq0MK9Zb4bel9FStrreCXM4ddPhq+Vs2AGgYsYIdoBpdwQ45Nfa4bIDXpHfMnBourAnMYVRtrgkMoWY/uLrSb0vplaMWSfmO9JHFGhWpde9RPEV1CvLmrBuDzz32t6VUG1ejQb7f+JvrldJTNelsYEAzS1WWfnxKH1wk5Up63e71F/GfqU7ATsQ/BP344a6+7eEqqyFAFUi5kl43M4hTGylt7Pv08ocJ4usbpQxGD9+2umoIWIVQqnVYGKzJIlkeqLwD+vbKL0xTbgP1ne0YohzefQK7TsBIrbeEd18CO6I0qiL4rNMpvUDYnXgax6ot+Ju2i6ovU90WfI2tcNf5+yXIvIMdSLwaFDOM8+CgOn+/kQfvCUxUNvb0oLoy8O+yULyEaP+UwM9TwJlvWjFPgNkvTptk+9VyyiQvuq5ksAZZKhxIFCc74Z+UdzR9wlOKXPHAXLVnhrkQ0sPJXTZSff7k+eeAjuM9PN/TplLs7KdOdsuMNtZDttnZT5/sWluRsBvE8GzMViz0kmjX0iYswAwvc2i2wIeXifwykAH7jOaYFQP29arziKKxGwgmDnf2fRgCOaKZ46I00xhkz34TcCab6/zdi3GMJMD9SM4jNA2A44yNEwEozoE99HxIOmUWM52oXk6kzFQnMVDEfJeZhJEqn0RlznmsjCJmEM8kAimeNov56Y6NUS6nplO6AbiZ1g2my2uysIxrshBHu2xAAmCkrWOpErEiAV5bqmVa2I5szi6plRhqyVXukLEpApAYIHoNBEaumooVf40nSpFo7Wp4tYQ2BwjT5spc6awbDMajXiQKUgCxq0qRx1oIQ8idLno5Tv7RAAFLK5QSrI6P8Grd+R8UtLzeE8vug74T4lG1hi6VLceJWQTrfIJrLREDHSAqO6sIZOU0XOyctHTcnN/y0qvNW5Y5m6/sK6UWLV2UCfBv4AL7CDsMwF/gHLQO1yPGviKgHk+5nJqOWDKmOXFcC17luSduYPAZxJypJYp0UG+2+XAP1t+HuPpc0us7S8aExtBK4OyuNxXSsa36ZXGMeSsoptF61IiVGEKtU3EN1o6prw5xZLglRJYtswNxCAA9JZjDKzG0ubJ0HB7G/wxdFdOBC+97QbB0Wbh0MK6IahoCTPpB/ZSWXv1wTz2B1A00rw20XVbXUXCMQqFUiLvN/1kiAYpiFW1EOa+bQlJubeotxyn2XDpuSHNEW9CmLEssEBprfLRMJASH5e32FKhUcQ2flGV2fhB8R7TlZLvJDgKQu6XQ3tRhEfEU6rrwNKuvdwRVYuDhrJDXp1anVvyTDU25/Le5kuZKwxefXnqnsQR18nBxxK6U0kOq/ZanclmEhSDzcNx1p4sAqhVYz+LICtpvCQUORbwXoJ4vVZpY13WpV6oW8dk7riRaAmESQmJ60d49DsAxRrSID1WtWLHGCSiIAA2jKY/TYaHTQdBBIhwAiwNwjOE+hmMv1GySam7AvqFDp3olUqlk3Kphm1baxjMYDY6FUgE+s6KqLO9vfO6ctQseIeu8xPIm+58k5kopkc3lDiQ6qB0ZaxVeLmMvW9N2pBQNtSqjfO4JrxtOxnaoAV23vN8zqQnqRBt3CBuhf/OyffkNHZjDHtYEPolhRrPoq0NouBlzolyWJbWHL8sMgx7bWFixTUeRzR9mtCy5UyGhjUkRjL7XsLFiHppOaQ5dG6rsi145zqDGwIaFN1OmTjsXplOaH8fxjsemPLGy2/w4dpCemoZ+0P7oxl1EEoMaU2zah00mPthwR8XWUKZDPWSGlazBUZA5iiyXU8LRi+0AHolmeen45grjQLU9HJiZD8XjHQ9lanWDEPO0yB3SKDC2e35cjcVnxTpnCL4Mjxv4KsSHgvGOx7roUq8UvaRxUGMhGmZUiv8Mf69F4x2PkdmUJbaScDaQlMoutt6K2GTj66/3xq5I7J4fJ0+5xKOzc7ssrbKbWiW31JJ8omVFa+SsmyKHRi8u+zgMsncaFM9Y8P8PqqZaXSDGyn4e73jkrFbkPDRN/zl8PymApb+4dBzbb4UXEOomSOs9aOclBj/up/lxGu94TIqa2WntDhyKBNo5EAIDEocgFcxl6bWIH3Oipu4/aByyzquEtltNf7jkLBOBMo5EHKo00rN4pjjPBc0hWoPIPmRpqiq41jRpiv+1ZrRDZ0FlRhT+X2tGPaTipcCx2j/Qf4UlvpXfFnymYgTFwNeXqW7LLs7VsoyuieqsyMfmfPMJiFhJWz++BtJuvFaYgSAC+pDVQBp0Q8jZlh7jDOKNNqHOrjWQ8jjgmL0gO8YQ3roc0Su/+tM4CiNy+JCo5jidMg1ol1VPYiNn93BCc9WeE44l0GHntMOjFpoIoMOnZmZmSNDCasySDp+amf3Ez6eGpk99618frtRYps1d6ZnXraSt13xEesc3CZuxTHo+cfqqM3v0hWODwu+jGOol0PPi+559Idmo72Od9Dx9H16oV53HAxmLsG/0vaZSVJUXaET4ht9rK+XAcjmAVYiGSdwVvteapYscVuaB5QlOVOhwe0Ti1sPSzXHVmqGDERHiFPNnGZyJUh7Ti1c4nhAdMdasIzrCzG7wW5AcOJqpTphhHv4ywRFpkdyFF9WQiYsYzLYmxEUOO70MsK0IcbHI8ZdlPS5NSIsnFBmOx1ZKWsySVEY+tlW/THQ9H7rbAvoyquJ5yLLkgxluBJYWuhcfURX/FIQm0gDOtPenplO6gVbArJXFogrlchkAImmXA5AGqJqrEKU1XqRI/AIJlcvz42E2pTdgYrfLKajswzqatGCq3Oj/JLkBREvYNpjzdEsCxC+Q0HNPiAU1CT/OtLlCyxeASP0bJAJGAJAAULY4ADnJgHHkhUu9EibyfqMsafGDpi71hFfrzv+g+XH0eHaet8RLYo7wau7vCmDZWXQhYdQywZCdpVKwwhrGlKm1vPTqFXNuOZuv7Cul1tQlAEBZkc5dayEFsJxaFraKRRdio46WIW4tO5xF5Zgrs1yMgG4KAM89cQODzyBGTC1RpIN6s82He6UxEVd/vScAMDZdBqOAjYwaoW9mwoP84dQl9Go50+IHYdkMsbrz42VfHeLISEuILFulWBkYAiBPqRddiIzCH0+hU70SkXSFLvUggqTFD2KgaQgw6Qf1U1p69cM98YTWzyTQvHY6kLK6njYHCCEyin485pq838SkxXpPbrBkBmU2UcTr8hIvlYA09RZeq5ylv5NSmsOGElPTBW1KMurnjDVephU2Bwhvt178IFVcwydlmZ0fJFbYBCrmVTUbAcjdUmBqOrBo+FOo8MLTrNJ+GVozGActoRC2iKdWpyVjib8Bv7i5kuZKsYZCBuCdxhLUxsUBbioOXLxNJg+p9lvAAXHosogLQebhuOtOFwFUA7CeVfBX0H5LiIzC01ILTajwgmp5t9E6cSCcQ6SREIIavkcDZIEtWsTHUDs4HWMIiugGqNsuaCKNJCAALBpAYXMfwyhqhZBmk1RLa60wNkMqomwOR3Woh5ywL0EwJByWqtUVE5j3DpHgJsZuya+kr5jSZeMa+7Ewym6zh0qNiWntQG0g8tgXZeyHYwiiVWlllM894XUazNhoPqCPbZHokANhIwzGsGxffqss5rCH9ddJYkzkLDoWEuqogUYCo4j4cWq6xf2i50obuC+2nTuy+WMilyX3gGa00QK1ogSGCm3YwFYPTaeEAzlkVNkXXQidEdiBDQtvpkw9DC9MpzQ/zoOzj1HRADE/jgM5TE3DoA3+UOxdRBIjsFNs2odNJj7cqMHZh2aN9Cx2mSvx2JLGCIVyG43AziNmLS8d31xhHKi2hwMz86HU4OxDs6aqsUuL3CENWWW758fVwKFWrHPGC83wuIGvQnwoHpy9OZZFl3qlGM0BR2AXSpik+M/w91o0OPsowWaDIF6LssRWEs4GklLZxdZbEZtclvi/5uQ9sVqhSOyeHydPucT/lQS3y9Iqu6lVcmMuySdaVrTGtsiatKdA0Mcx273ToHjGAm9KTbXwWDptruTB2ce26KFp+k0W+kkBLP3FpePYfiu8gFA3QVrvQTsvMVJ7P82Pi8HZx6ComZ3W7sChSKCdAyEwIHEIUsGsYUYJdF2YNHX/QeOQdV4ltN0iEgYkDkXKBBJmqNaXqW7zwkocQI1yOWHsWZsJaxMzV/ECURGjnQd+MVVxFp8RFptpHIYX5hWLFHTa8MK78tsiG9YVNJmuuqx6kkh9Mlft5VbLvS8JPRpW2PeNLsrh/k+kKu6Nbxh3MHSM1uCILWVZloqBSQRSBbQCoCjjhtzO2qmosW5B6Io8aGp4PJpZKVDGz6w0hpb+w3gBUKW/jUyldQtDU4GAQGWl1fDbyFDhKaoHcQMGlZVCmI0MEXrVedyAQmWlzmcjE83hUFmp69nIRHMoVFYKYDYyoWrwrs6IAiQqK3U6GxlzvZdBaiYNsKis1OlsZJbmnwIiODQqK3U4G5ll6dX9Zeyz4wEsKit1ORuZaLhQD66GRGWljmcjY14cwaFQWanz2ciQ8Y1mqhNOmAgEDCordT8bmeVeJIdAZaVV8NvI7MVyyFVWujR+LdWvrZmHaK+ZMYrbXXxZfi1NBfhH7UYelt9sw9yJNIBRzWQ1gNd+TA50vfCoxgiVPjpcJircZvSDQx8znclUZqFaOET+FrMRv8xskxlFb57yKpHMkpAYJqs2o5G/F3aWvx6222aVuh1aVmlJuhj2SdlXoxlPTf+jlZTmprAXDg1dTD1t5sQL5KY/ncmjcj2TMuCSVBEjH2fQeg/HIu0uHRdDF4dkHPwC4Eupq3NpYRsSWQTMA8vz0SRGPibNlXxcSwxdXGYEy5kAgRAKgQbLkYTFIsKa4rmgkY9ZWCwDiaGLcUjjIHgBFA3HZ7LVM5m0oJGPoySGLhYhqb1A3HQml5Z6JlMW7BMr2JRwwBxHpGYtbBzZ6plMWKiRj3N2Ra7P3UeDxzgDHA9WzthtVQ7MJWohRz4uXYmhi+EFSvgxsdOZPCoPiUy3U70SwfDIx7e5P3Gj9MRDFz/3RIK0k04v0IqZzqRtSGRyNHnHiVqaEClm2QY++8dVkHKToAZfZt0jPlXR1D0IgcE/roLUdHoeWo4xTBaQFGHoKfKAqAhCvTS492VUxfPiM8Lioh0+pCcttos+wNZ1vSm69oucg2NZEfpqRUJXqytmuBRYZhBZ1oS86tvCz0QOjKmrZzL6IrQLt3I3CzNcqw4+ymFbdfD4GumZ+OJbdfDomqrGBrbzqrakxeBQgviGawdDB6ALsU6nTAMMy0YWjXUYVh18hHNs4ld18DHOxMSf6uAjIgZVB0+FY7MqWi54Qsg605usqg4erXsZA1juKiFrR8+qyZm+6uDjq1O9XqiyQouc6asOPkyF78VV1qcbfEBgdfARVoXV2d83P6CvOvgIrN85Nj4gsDr4ANUGrmFhjiVrfkBfdfCx1fAwlD6grjr42KpKDJQ+IK46eBJcVj2RcmdtpThULX9wIVbsqQ4ehYcmGXdTGwiyqoMPVDsB6ymntbVTqqqDD1Wr4LIucl4YZdXoyx/1yFcHHz2Zd9bCMCtGSdt3He7VwUdl41wtTG1fZ/8tMI9jVcBfoSOMpgaZvS99lGHLgx6lio8X9+hrZn6UYctVMLt36J5j8KMKOgxcDkZtgPDCVaY+P1SNVfwU4YHKdJu3AM9IQ7OeonrAC6vA5WAcLPd/CpQ9BSlqLH6K8ECFEkh5SFavOpd0FUcWhSKupdeFaRyrIaX5HUFi2u9FTxG+CVTrjZScKcLvgdd4DNp0alkfZ97wkO3FThG+manDM27UuMhjSqSGg8HBX3hk4mahU4RvEkHZN4nuhGNQNi1v0tzz41gNX9QU4UHkOG23M0X4kb9aYXKpp9Vd73GFfYFThDcEljNF+DEs4um/U4V9UVOE9zbeFOHHmGh6LqgPu8W032FT5BThOXDt0qYsoTnY2BOt9+i0OZ72O/i9YqcIb8f24DQ9Rfg9rbqMHTVkcdlPN0m5iUhJjFmVi50ifOZBkr0pwg/BmqlOeOE0pMxRvFpZUsW90yS+2CnCS2rZIn40QCpNuA+hMyJxsVOEd6hFH8MxKWpWZxqLINdvF0BZ7p6WMSBSIyriKDZMGzsEjdIp0+DFidJGoqdQKxbJkhFJx4vy21AW5RfxEy/ewY7QY/7aIeVXu0iBA+Du/0Sq4t4YZ861Q3diKCbUCRiGvix48aBOQLyd6hVOmAQv1tUJiMMMfyQd0XUCxlkk1wkYkdFXJ2BURl+dgMFqHeQIrBMwTPWBjr46ASMyAusEjMjoqxOQAtf1ZiTAnzoBcZiPRiRAXJ2AAWoe9uiqEzAC6sEdY0hWnYAxGlV1AkZoyNcJGNfhXidgoDsQBs4fvarQqVZ8QAx1czmSvQ8gQf2OuSTa83UnKoloQY917bIZVZXMF5BmU9XYRDSUoLZWGSXaSVTVfAETD+gXvTao2FeZHGZ1zRcw6ZDV4AoweAJ3zeHRE3K5XF3zBQx3uwIEBlEXMxpR3ZV5jGLR6TlXy3wBg10vQ590oVPynA2eoGb5x2MUi2Fkylwl8wVMOuCQC+s9eAEaUg/HZYDBR6db5P+qZL6ASYcmjM/Xnx8HdVkJEwh8aLXMFzD1YP9tV6v0Nnyy2FTNfAEpdaz6hCIlUmuCk/DztrMsHQeRP+wqtyZ2V8l8AWk1UZ3xwkQauPo03aC4UgxxzGMUi9OqZL6AyQdssFDGtN9aOl6q9lvVMl/AtEMTIkI6TzdvlyC5WS3zBUw9NEWPwabTYVCCNKtlvoApAZIiEH2IOCAqwtAsvHBtNsbxGWGBtEuq60TKXLWXXLyrwSayYV1VW+HnOPhZ1oS8MgED0DT42VaFthhHX4R2YWQzu/izp6ge8IB5/xAFmbgvxFivOpdcnHsbA6Ib1r0xFFEN+44x3Dzj+ZLgJ19MVZwFojakMG5k0SiHZ9XBRzpBXR38h6jekFxBXR38THUiuXhUHXygOgsuHKoOPlhtEELLl3GoOvhYaMqj6uAjITZVBx8D8ak6+Ai4YFR18CFqFVgYVR18/BPK1cFbUiK52FQdfAzEo+rgIyEWVQcfqm4GFw5VBx+s2uDChurgYyguVAefBHBcG3yVOtVKD4lphdjQtArDVMt3F1w0/9tGSET7eY8BVn3wJNsgaMsLZwGlw4loJlGW1FihvA5Dd4OAUh980FPhQb7QG8bYsByyKsUtglIffLIhq6EVqLcNtK8y1uHHhw5SvAelPvhkgOPa4LHF+o8bo1NVGOK4uE1Q6oNPBT5xXBu86P1XqDDE8QCQ+uCTDegY13smHCPZVGVAvuUSSH3wwW4qDlzXBk/DGNv8/8rmCEp98MkA17XB57KEYfSqLUqv10OoDz4V4Lo2+LIUpbdKQxwDqQ+eVNf1ZszgrjZ49XKlB8UKQxwfLYHUB0+qVjcIsclyapFoGGPbOjzjyAJQffABb1sWuKwNvqIW8cslqPrgw5153xEErmuD54GJKwxxDKk++HiKZdXBRzojns0dNM5pgbv3ZVTF8+IzwiIIHUdNz9cdTphIgksT0uIJKJuqxgWlIK9YJFuC4y2KI9aiOaKsp/Qex93OvMEi9/k2B5s+4NEdF68w672OQz31+2afxDGAbobf0erK/9KWEXSsOtjtTPyNs9d7JpiHpfheLDyGJqozRrgc8MbZ4ygfP+f3268QXubUyZREuQQxQCkk3bC945HqS8shC17f9pJSv9CfANgyNXtyrwpkpLDVFQ4pNGDVlzgcslBidAIDEpDdKJCTljCQEa2uMNk7HqvA4ZAF0yA4kIBFdqI4zJMJLXGmKzpSuNIzLBKgeohX37Sqx/7f9pHq1OgE+hNwab94pcuFwij3dzQhiesFwIvtVUri+QXQF/e0xKke0pHCQz1FIwMHiPXN96j+BQbd8sFYdMI11Wt+3+51cTHsq4XNl/An5qo9Lzph7v1XAt9/K3C+EWqBx54AfOgck+ffC/oe8cTsaLqiZ0dT1AKNfSvZ0DkmzzuBS4fu5IUV+mkfYsyVyAsLL+9mTHmHZLyHaMKtn2Cf9zzkk8yW/7ryJ8bdTMQAGrRKHbI92OUN9xx2XYnx0ihX/jukDrEhykdT9fw7Xv/Bs0FP/frp6QDqKdw/PildbXb5oPjhZ4q7/sMvfMr0wX/kIz8xaFT1yV2J0+AHBv0U3mxXenblup5u82XxfOI0MTvaquedG4aMH/5Lxs8+Vxrn/XCYoBsoo/knRt4wVKSw1LKlmSDNDiM0CSXPOxEI9o295cu/T0TdZXkzohPNjYhAKEihrWWMWfz2/GisnpdVuvg31pYzjfg+miWoRy/chd9dHXmAS3IX3chs4R1egdD9ahDzBr7h95rolhLuMtTjQZ+H7+6O5UQluUVjtBAPWEXofjW4mfOJ9r1WOvqiuPHPUnnRQY270icO6ugYyCR3VLQwhwDx5n5VG8m+PV87Q6cgaMZRedEdtEDjI9JNPAeRuegseK/duE7hqgUYvvyBw6LB19VIesdWrdg3+ENBqJsrQgFRq/hE64+n0hCT+k1SNsBug0AlzgsC1n24hG9+VXvISLeCCoDbKh4iARgqub3qocYQkVA15atXLXtoBGSY5PaZXnpkSEgXX9GwpBiYRCBVQCsAxsqo0bbUEAwNJhFI8bQNATD6JwhbAViAkL8GNmYQ12hv38adz77z22u3f7XdAIbb277jG9tKbDfau8qztnbn92yfsd1otG/ZJgLE0R4yIAEwbNA+24HYCkACiPw1sDGDoAYm8vjEYU4w+MRtW71zbfsyhxjUPnvgavCat6zYNt8n/J/3nkFiAGGSBxDx/uPhRdLgGxJInvQ128HYCsABjPw1sDGDnkYbU4M8PnNs2/iOuXc9bFf5zm9uazGcd61putYEkr37EOnsp6O89wyQAFC2OAAN9/3HxYujYROkBZCtABQAyZ+rjRk8oMe/ZY47n+dq2wDWPr0N7wlrC+2vFVDYQ+3fe/vce/eBx4NrGa47+04gPwugDTRh15xBCuBQQDrZDnl3y6rNwMZGsWclU42cqAzvpMGBZQR953r++trsA2ydsACxFYAEIPlztjGDhXDpwlU9ZGXS/Qeu2mVF77+2cGh2W6vmrMKFMl95lmVIX+utAICCiw/QwgQBYJD2BhOEhrFR6FnZVCYPMrzgDS2vieRkINu7ry3YrsoKKFsBGICSP2cbMzhoQKkUPBW6/jb6vXYb/dR9z3w6vk0EMeLagrj4QY32tWevto0JuayoKwDwXeQQGAVsZBQ9yFRNDnjIiFSBNEArNmkBYSsAC1Dy525jBgV45eCqgvNpmNcKJKYDfGH3h+t9VoXbjoy0DW8CC2YC4hAAeUp0duFih+PIKAjoAcghBxA+DqxrM7RgM9cZZSOeg7EVgAMo+XO3MYOE/Szw+EZy3zNW0mSg++GtAD7AgPLq7D99NiCCwJFCLqzNbm/j/kDK2u/pWlOIjcJEpjK5rzVMIMFW+oUdYy3PwdkK8B+Q/DnbmHmmsUmCweLAVX3N4SLC1RWZsZ8dVRuQvtadz3yA3iVmdx2PBdPs1Qq66G0yaj8QUzYzeeSmjbwKyVzlq1IC0FaA9+Dkz83GzFVzk8R/15puKbbdFtdJXFi45lBH6H2AYQnRDkHASCYQgCzGSjXEq+BuRe4JDW+gBJu4LABtBVz1fQEnf242ZsqbJN5fTXzKZwAWXkMBKhwHhdfg4SAjTFDD12YsLsTCNQ1IayfbRi6qGL2KQRZTY+6yqUzuiSobybsyW+liMQFjKwCHBYj8Od+YuYrB/ltWB7VrDfVJK3AcgGKq8OAgu2rcyHa/ASA7QZjLr+CjViFhFHpnMlWQewKq2QfgnwVbV1pnOQFkKwCFi2Hkz/XGDBp7xBu/QJuP2A323MSBcA6RRkIIQo4GaAhst/GaZ6pkdOzkq8DMOFtxHZCtABR2NIWRP4cbM6gcY2iosMe+9uC6Lb9GEhAAFg2gsKP7GDpdC53jiFVbJyuAbAWgsEc8iPw53JhBZdSGEiwH7BMWMLYCcDgYGkL+HG7MIDO8zFXjKUukVdkKwHnAvmpbM0QP2Gc+esu0ClsBSI8sWmlrhitjtcf50qLK5FRs9bmxPDvU6noAjIN19epVrIdArrQ1wxWR7yMF1XJyRaDLDc/ajSFWK77y5/nc8wH7/EkuIRrWRA2hC/ncuIHN/08qtgY2J1lA/FEiefR87jePkkuERqNBajTsMfsANkSL+cACkz0C134bm5XAqc0tSn+qM5ojn57PveZTchHi6lrYVh08PwxDcd8znx7bBATZ/DFI220pr33JKI78ej73mF/JxV8DxyL+//d/9/vhjcA8IKp9VrSudQYnRLYG5ZhlH/C9KXDv3V4G44vGvKM38uz53F+eJZcCzb37oIsTjP5pbwHQYVmOd4hIYnBCfHsxrqAG8XFmEKPo29QY2nTIWb49n3vLt+RS4GQbj23T8Dl26emt4eyBq3LQYsYx3ENwzRvMB1L4ord4uzGkqeUs357Pr/rKu+QSYIHcIfXmbrfve2bvPjWmjuWMM5ROAx6fjq8CfLJ4646/M3oj357P77vqKf+SS4O3Xm2Lfk7e4IQ8aLEUZ4Vpv5TAJ11rjuLIt+dzb/mXXFoWEGPb6PNOZkBSau/HWHtUa+PZ87m/PEsuDUTxdu8+cnGzPMgqXuRGQys8Ftr8eazmU/ijQLthPp8jtPAsuXQ4cBUHTT+LoxmqCkNv0GJ2g2LoHRrEEOJ1ga9PG+KkfOXb87m3fEsuFeCzU7jqZ9cUwOwtbc4+IAYtZgEHjE8IH9WKQQzP8kewC7r91qiNPHs+9/dq3iWXgD3i8U1BNWVXJFB4JQQBGPfbdigct0X86I18ez5/pq98Sy7+xxjKLgCALYutbfj4lUgEYNzvLyRxnD6Gozjy5/nc8z3ifUsu6iq8ZvCJxsJgmzKaI4+ez/0+xtCz5JKF3gzCY1ijK5zlz/O55+Ng+ZVcwvB7xfBGU85y+HyOK3fJ5cm0sUfv5O75HFl4JJdtFUMgK9pMG5uKiODOCXwe4EX8ql9VE7ZFOHwrvw1T73Qpi3+JlFlBvstvA5dzX+KplH587cxlGU8CXtAktKT7UR48TeKxVmYHH/J+F5e4RSCwHOcCXnJ9Ms5eRKnaUF/ebwVMiXwty6x4jxHSDz0vTJCGXnocvhMQhzXeR4k1gNdpsXklW3UK9MXouGKcXb63IrtdxXemxPqmi9DwVpsszEo1QL4YHVuMXb5bNYfdotp7fQPhbaxB5pXsckktAF+MFkh0vf9HRpMrvntKo4bTTXXtPxS5IY/Kb/+RHg5ntqUmwM154SQUHKuWmwqt0UQAPcbIastNhdbSJEGOVW3N0LbURFlUtbZmaGttwtpcJh3rlUmrenxtg9Ro7O3q2rFqOxnpKAmyfWO/M//y253Mh9i9x9vyOCNPuDEYsUzm5cIlUw6Ai7vyLird0E+68uH7XSf2izw4hGI+MDjqgf/faGEafh/I9gGRii8Jbk+h3JCNBJO59cD+PeJtO/0Oy+qQAZHG4a0vcKADrvWdaXt+XHQEqmg9OqHEyLbYu6Sr9g7tq15pz3Q3LqPoT1ynQ/YZSpl0TynbjZQum3jQBYfWNkaacMJuqY2dmNBY2pRARigGTbnFKOKdvEApisZocYocB8wKHt5enGLQ3OiQikWOc/KiO+mRitXZByOVj8HAzLHOVJkV003VrnnsPIBREQlMPbYBfh+fIgK6yqWIgh4ZHiVRkKmJBnsm2+R4J1s08IhRrCExYbnvNJLu0gSEBwNfxzp5kS3q//AXK0hM2O48TaS7NKVzY3jyOCcvio0TQ+mHv+46fkhPSwxe6mrqPLiTWWfoun5wnGEUUymLtpnu7T+hSGftd1iEkqh/7xl2Xu7r1w5VJ8ernf14Z332ZQdr1TuN7J8yOt0uFkyxtqzrccz7ZKTluNQ14ebdLiUj7rLGvPc03jk7+0iCuYopBiYRSBXQCoAadXCNCi4HPGuM9LUuM9r6blfs4X5ORrzG3DZsbW8R1MDZHwqGBpMIpHjahgCozYJYuxF/OeBZuzHCP60bWdqGPHw7JyLKGuycUosjm/rvGi3kU8Nmgk+TJr/aboS5tiJAHO0hAxIAtfg4tNBuRF+OQwu4KCAjfK2tbTMesZ2TEA4c2FokiAlF+1OMbpgT3LuP1qbCHGJQ++yBqzBxafM2wv95kSiJAYRdHoAfoyp0hzOC+hEXJAYU5qQfY5mzHhk6bB5ONTSlpCm1zXnEdkZA2DR5HgUHLYhh0WtotNt6gveNtkywkg68PCrXmqZrTWchUM5+OsqLREECgJBcA/UVAJQGozucEpaxfBCfBIQk3uWV8ohH4lpbWoI1eh4Fs230wQ3GqOjRwhDyTW+92oZ1cfj04D0+qLm2dtJCzQVYrhn2Pvv3/pegQweuiqXVYd8Z8MW6O6j1dBAAtBAfwW2ZRQttCGPXRNTXMM927VlheIM4vSsQLsGb7of1/NBmsKkOCdi19wvtkoliGl+OwM7L/HBSrhHI8F6+nsxkZLa1V5AkTZ9H4bQ4dKfZNYURkWp4Kw9yrRn8kTGszT5gzu/Q7LZWm+hKzFeeVVpb+FpvBQAUFOlg7UKEHQKAlzYmCK3DgBW/oDCFZrLhgtO5Aub1FgwpwgwE/lr3nxQkvCYZ5jTbDX15eYOgctVlhhfyrrwpX89lsh60TaxgfccoRaAoCHoBgVCZCYJVMVbOWrgaP4/CpYbXc7t3Fzcm1vjIGMk1msFYsm2OQxy3AO7pvmc+HYPPIOZcWxBFOosCrSQGoMhlFWgCgBaLCC0T69wjXJc5sk9vh8SGC059BdT60Wwr7JtBwMEaJoHdwvE21OU96yxFzewDch2LhhPyr2C8ztcYNfXDX1LjUYfSMR+FfJ0R4AsAQoUOTDACZDGeD2wpWKu2kThV7yNO5q2gmIY12+hKxHFWLAI2WGGes+oQRw2sljpk4Z5YbSMCQE8J5lD5KryAldsMV9hpAGy44HSuAOEcOvAm3wywvS1JQHg1cKmh6PLqQ9xLMqwQiBI8PXr2dxbX2iKu77QNxY6hxoMOpkDmopCvAwJ4AUSozH8JRoCs3PMpswpVY4GT/Sx0Ja6I6n4IMOEH4drP/NNnr6onkPYHmv8faPez9nu61lT+675nbDfWuKsNG86cngxL6lpTag0TK26zX4lN+FpvFZdk+KAHx1r1elstrjewDXCqdIWh+mPOy3YA2fquuKHlRAgBA8qrLhtw3Op3eBZADOLWKxuSvcDjpxlGRqkQnLIJr5iU0ivSFxnXDwZOMFawCU4Ba2ahLQRCZIOk1l1MJ+6irxOvidwcuFrRRpTz9rMTajfAmd35TF7/p9kHoAadaAvatC2dPQC2nBSOLGITo4o2kFj+7mHU5veVIdHu1LVN5FXzDqWpe4MbtOgPwsbAAQoQJHBkUFCTCQptIsw0tLsBDPQt3khiNuQk29hEEoEQ2bHaF6MYvy1O15rebk+BShXX8Em73fB+kCVEOwSf9JEdBCB3S4G92MxBnBax29X9YKfa7QaYvr6WOE153f0mVc22b/ix+6DSQwcuz/5OR1VxGKsGkHM7fRr+joQQvk6rZIU2Ck5JjyMYSQABMHKSbeIngRjEyOC42ldDMGJPo4/qDlw1AKuvd9wSVImBGzEvc5ZanVrZCY4Tf2OMWHQyfPHppXcaS1CLRp7g3thwcZovNCHCDP6U03GZJ9GT46Vt47F02klhwtB/WoWs1iwTHfdx6lw+LDQGI0oD4CcQQ66/IrLp9K5LBhJS+qJXTQswP+axW93emNLdxoFkGGYeQ7QBya/ab2K9NcQ4/xwC3XplYPsxjwVOcLcEe/pbDDj9vPTsq3YLxkLYaz/pZWJmyYPhItXefwu9iG+/BVVL31213/IkGziBzAFx1539qusOQVgFfyXtt1SMuBbR6ce8rNt+yxc0uloQWEoIPPuAdJlnrU0sOV9wug1xFc5y+62FQW6/RdS/umWioz7yr8NHhWacNrcUodMvPvue4N3U5luAAse94Alt//rBQEJfkhU3/DDiB191fbcFHjwNVjJsgAkQgc8cI8AsBOYXnn68YiH4M1Nzj3ioeAIAOK+hjtNNxgmESSQSQwBYHIBjTOw69yhDHE4CB08ZjcVGS1Btl2RjoyOu3XBBY2/LRMes/PQmJEhf9Od8UdnrRV93n+bjAl04FvzaPUGSFXbj44WngW39oDlUAxWwFuaCjV/3IFiB4AOJDg5dgqojlOhj+Ig4TnfCYxAicZAIB8DiABxjKljnHsTxOJ3arWisRyIumbRLs7HR/rWrfaFzlyXp0B1nmlqE4taDvtDxyd3CA4rjLIAECpBkVUKXid70cfgYUsAKO81ufnrvF/25oUyPPELavY2gB6reSzPao+AALVT0dW8RSpiU7o0VRZZgJMMq3ZvSs7t2CaaYe33xbjuEJCoq2708Krk0o5YyzzZYGzq5gE1gwhp7ActSpnyLr7sr7OseHKoko7Ldy6OCSzNqKfhJUFf1Yx5zd99bkcAL3sOSrHG7A9fLPAf3eRphpcypQlV/YPzIL/pzVStujdU+AhDB0eDFjwf39EV/7oWnE4s+AUysu2MEZFjNHyRZTXc7p8FPSeG8ewSs2hhYMj96b8LnyH8jGUJyNHQ18ILmpdKtbpHtt4Dvi76rL2yKBWcGeawvuu237gFXaSYwLAtQD5qb5bPR39F0NEhUx6qxg6HvYYIIkGOC7x7bL48TYIHMg3d4sFAXb8JpwfpjtiX5c6eHhK47oz6a9lTHaWPDyxxjgggQ41NMx9RxAtHIPHjJQwjHok0QpoeCITyvBc17z2tGd0R5NDdgHxP4IMCnmD5cHifBPtyDVzwCLdoE4gh/fgyOHhqGlxlWit6qnuiH4QjHHKWO9qv7i9GDrZ5QHWe4KGJk0orV4tjVh7yc6R69/wiPFZpElErGl7ApGkDvP7ID5bPf4P+N2DkbY3r/UR04n0+Qhh1NWRX1hXIgDftDdyIGOue8gKqTnllxEswInkUM6/5ihFhVW2KAiijj3V+M0KoTUApQEmgOEpTZFWOZFqY/M1Lg+S0tMY+6CJ6If7RFcGkjsRbFxSKFN7mNLf9t+U8gJU9hkUBFSRIVhne6uDQqfvjj/+NRSZlKxeP/Lp2KLf9t+e8D/dfW7m496mtqOf1Z1lf8/7/9/q3h9Piye3z3kr/x0FkikX4eifg/uzenWk7/i7Xr7KmG09Mx6ICVRFIk4kq65SbQf67N9BI9UvoKf03qZSXfXLkm0wbpMH/lZ/yuVDon0C95H7eCoLL4Qqnjp2sy/cQw9/kZYulLPs77eCBVJT+0NjfvpT+uavJXkx/Oz/q8JNOzPI+Kfn9Zk+ldrmpxshcR8rLiQdn02s2R11FdrsEUoRqcPs+eWr3/71/NpYrFc/LdHUgs+EsEire6FAJEIzYajT3f4lmO7wbUEAso5KM09HqdNBikYYh6LU9Gq9XpdPgF4qCGUB0T60ZQ82q0zwLVge0zSrd37c61b1kxPGs3AOKQIdFC6BBUu6FXR2d0Wp16vd7hF4iC6tSIMqUivV2uOsEM6lNKj6uafTn9+mNeOIoUCouPYyAHa1C0d1uJ7XZ7P7jBBh4WOEABwlFgMiFA1HozNUaP1anvyn0b8FLRANapX7x83Qfs1K9M1js1oVxSygU6uzJXiaTrujfB2oE8KBpLmNNDALE6Dl5wKrD4OAbysAZJ5vZOWpgJf0dCCCkDQwFQW8GIKHW0WpACJjVoODFgmBQsRLSPAToyDbpyuT2QILUKc3u5XA6I1SH5528mFmjmtFrfIgRQ6w3iBKcCi49jIA/r2OBpAfYzALrB/WtrB/4667vfv7Z2so10h9aMQcAYViBZaDdGp0UpnAvJxCyIYo5RogmoKtD8eFmWU9M3yqzjTzxOJBGblpsDRKtkOEZiDJmK1YsPoke894v+3I/5Zv/wMVprmXuTAQS3eavbGzEc1cEpr8mMWoKZb/F1f1569j1A++Bppr7n2GA9+fR2hNpnZx9oH1pjHbgajjsJOrRmcJ8uYKBIBzSNUWkdefTdA4qUVuv1I+dSSpN23uXrJ952M1I6f6a+mIyoEwANydA+dSuQGNtkvVMVyrlcHhDqlVnFn5mOUwEpQZgEHCMJDAlamI5BCc4Eq+sU/N/jpxcSLWzRj3ns7qAAcC+sNQ88Hj9tXLzKRVJv/KI/B7RmB1Hb1+AcsrZ3W4m+sOarfZZ10op2sw8E3XfH3IHBYlEBcyi40gY8HZ3WuTRZv2I/b7JuLwCbE1tWxlvc2AkPWKv1+sULDBGOWwTXGfZ3qkTPPRG+lcsppTlDCASt0pg2B4idbCsQkJZT0ykA2Hnz79Pbwzu8pKX3iVWWXcMsMWxtWXHQGOempiGCLVSPC+9nTg2KdOY1g8t89j0QiP6YxwANQCCsfOFpwFo/KAR8SIsvQJDHBmVzaO3TI2ReTx+CEEHGACQSZvaB0evd4AE79cXLf+b8GYgxSat4XKd+5fyZE2+bsCMmwVvycZNVInwB8IqbA8SNbkDophvBac4FjPVe2Q37p6Zb4DZts7licSrhzJX9G8xgjOY6u2F/Hiw9jo4vPDsNjlF4wNBP0BiD00Og0wAq9TjiGJ6gHhw1GhZe+trv6K1X23aekNEJmPZ9z4xi70ZXV7+ysRO6Fb6UrFi3MVFHVxjSlZRMkwBSXTKfZg7Rgs8dXtZ7Zf9fz49PXeoFzZXwd8vm8/AJwGQ6b3np1Q/3AA2cJR8Xvg3Sk292kJQwKWF64WmDQQUcVwEbzmHqwRIEk4O6sbMtOB213j0Bmnj9m1NKk1oJ06S93Pmdic9Vs8CtQeDY2nx4p8tbjptP3FxpsVNcOl4KtUqsTkNfaEW8ZGoBB2mwDjmoNpBYVKlv6cX1g0VsRNvVQdrs3Qev8LXud3fvd3bv3ad3CxhuMDFqLPt5pAt1pXpIFzcm/A3vPCJVfU9A5v36c1M3QX7zw73+jVIpBKAppZYWee5WN6Wl104PnsDP8Q/C3S/abmPREk0mjELspqozkNqN1IMm/KTwvmf27rvvmTVxmh0rTwOIBX2agIGdVuYbRRbWwUMNGOzGBhJAMilELbnsMAtGryRXVXUIb1rd+R1BN8r+DrQnWkpwWNlf7/kbdokxGpSGpkHrB8VpUGZTu7mIZseJ0154+ov+3AtPJ9Q9QPvC0w8K6vBkkLruQFFsbe2DKmy/tRDXfguO4u4/o8c6sXXBym3nz5g2Jsz7Qbntim1W6Vj7gBEKd+YNT2yl1aqU+WjHu3tTb9lOzJXdm4Qu9fSmxFKeEJ6XS6s1s6fODNkrK+gVLo8FAm6/9ex7xMaiUqr1Wj/I7begWJfSrW5vJN3jtN+CMwevM3SDCSJAEE81mW8oBuHdI77Sv04pWXv4oJuNz6VEVWHgEdPl63T8YkrJnOiiEQVPWYUyX3rPfBp+xLic5seVsNy2bJs5PrdrKP31nS7TRpueMZyLg6GhfZbTIt4CSA4e41rEM8hpRHNw4PlgDi/DBBEgGoGOEz9JdI8xrPCL28GbJnVz9w4hGbiHwhDiuGrqupPLZW6QtfQ+sdBnqg0wgUdM8795BTG7KUEjCAPq/+vvDOVieBmUdQ4ULKoTYWQfQwa5B9F8nGN7jgbs416EKNmBUCHtc1EYgo+rrs7QGQngBbIiEMfpF1DMAn3IOi5dDNgHgthSsYhhGSJHbVDIgBaBs+dpZFFUBGTtDS/jdBpsCgJ1nHwBySzQh6zjXoOG/e4ZiPZsDoEMweOQPQTy0N89j+3ZHKsdg8dhgCC2HOPzn0okVafpSqhOfJpgnEre3yigMzmP3/UzNI5OziOtmO7Q3VNzXfU2Ra94Gc9V/YG0wt/9n9cTSENiSo4YRMSYkiMm0op45Mfh1ccdj+Mb2eDvAAA=) **Figure: Device tree overlay on QCS6490** The build system uses an `fdt_overlay` tool to provision the out-of-tree device tree overlay handling during the build. The DTBO merge process supports merging the DTBO files listed in the `meta-qcom-hwe/conf/machine/--.conf` file. The following is an example of how device tree variables are enumerated on QCS6490. See the board-specific machine configuration file for platform-specific information. # List of dtbs for corresponding supported qcs6490 platforms KERNEL_DEVICETREE = " \ qcom/-addons-.dtb \ " # Additional list of dtbos to be overlaid on top of base kernel devicetree files # Format - KERNEL_TECH_DTBOS[] = " ..." # For example: KERNEL_TECH_DTBOS[qcs6490-addons-rb3gen2] = " \ qcm6490-graphics.dtbo qcm6490-wlan-rb3.dtbo \ qcm6490-display-rb3.dtbo qcm6490-bt.dtbo \ qcm6490-video.dtbo qcm6490-wlan-upstream.dtbo \ " Copy to clipboard **DTBO merge sequence** The `merge_dtbos()` task in the `image-qcom-deploy.bbclass` class merges DTBOs using the `fdt_overlay` tool. The `merge_dtbos()` task iteration occurs in the following sequence: 1. DTBO filenames get listed in the configuration file. 2. DTBO filenames are iterated through the `KERNEL_DEVICETREE` variable. 3. For each base DTB, the DTBO merge task overlays the DTBOs listed in the `KERNEL_TECH_DTBOS` variable. 4. After the overlay process is completed, the final DTB output is generated. ### Kernel configurations The `linux-kernel-qcom-base_6.6.bb`, or `linux-qcom-custom_6.6.bb` kernel recipe file uses the following configuration fragments to build the images: For example, `linux-qcom-custom_6.6.bb` uses the following fragments to append the kernel: KERNEL_CONFIG ??= "qcom_defconfig" KERNEL_CONFIG_FRAGMENTS:append = " ${S}/arch/arm64/configs/qcom_addons.config" KERNEL_CONFIG_FRAGMENTS:append = " ${@oe.utils.vartrue('DEBUG_BUILD', '${S}/arch/arm64/configs/qcom_debug.config', '', d)}" KERNEL_CONFIG_FRAGMENTS:append = " ${@oe.utils.vartrue('DEBUG_BUILD', '${S}/arch/arm64/configs/qcom_addons_debug.config', '', d)}" # Enable selinux support SELINUX_CFG = "${@oe.utils.vartrue('DEBUG_BUILD', 'selinux_debug.cfg', 'selinux.cfg', d)}" KERNEL_CONFIG_FRAGMENTS:append = " ${@bb.utils.contains('DISTRO_FEATURES', 'selinux', '${WORKDIR}/${SELINUX_CFG}', '', d)}" Copy to clipboard To generate the debug build using debug configuration fragments, run the following commands: # Set DEBUG_BUILD to 1 to compile debug enabled build source setup-environment DEBUG_BUILD=1 bitbake linux-kernel-custom Copy to clipboard For more information on different configuration fragments, see [Getting Started](https://docs.qualcomm.com/doc/80-70017-3/topic/getting_started_chapter2.html). ## Remoteproc subsystem The Remoteproc framework is used to load firmware into other subsystems, such as the audio digital signal processor (aDSP) and compute digital signal processor (cDSP) on Qualcomm platforms. The remoteproc framework is divided into two parts: - Core framework: The core framework contains a common logic. It loads the firmware and starts or stops the remote processor. - Remoteproc drivers: The drivers contain platform-specific operations to manage the corresponding cores. A remoteproc driver registers a remoteproc instance and a set of operations with the core framework. There are several heterogeneous remote processors present on an SoC in an asymmetric multiprocessing (ASMP) configuration. The remote processors run different instances of firmware or OS. For more information on the standard framework that Qualcomm Linux uses to manage other cores, see [Remote Processor Framework](https://www.kernel.org/doc/html/next/staging/remoteproc.html). ### Qualcomm remoteproc support Qualcomm has enhanced the remoteproc framework for peripheral firmware authentication using the upstream PAS driver. The Remoteproc driver in Qualcomm Linux implements a peripheral authentication service (PAS) driver, which is a TrustZone-based peripheral image loader for remote processors on Qualcomm SoC devices. For more information, see [qcom_q6v5_pas.c](https://github.com/torvalds/linux/blob/master/drivers/remoteproc/qcom_q6v5_pas.c). The remote processors are used to manage the lifecycle of various co-processors, such as aDSP, cDSP, modem peripheral subsystem (MPSS), and wireless processor subsystem (WPSS). The supported co-processors on QCS6490-based boards and RB3 Gen 2 Development Kit are aDSP, cDSP, and WPSS. Note List of supported remote processor subsystems depends upon the SoC in use. **User interface:** The remoteproc framework tracks registered remoteproc devices and provides a user interface to boot and shutdown the devices. The user space can query and change the current state of a remoteproc using the following `sysfs` interface: remoteproc user space interface # start remoteproc: echo "start" > /sys/class/remoteproc/remoteprocN/state # stop remoteproc: echo "stop" > /sys/class/remoteproc/remoteprocN/state Copy to clipboard ![../_images/remoteproc-framework.png](data:image/png;base64,UklGRrA3AABXRUJQVlA4TKQ3AAAv3gOaAGZh3LaRI7P/sjfehrt7RsQE5EWag8Qa1RmskYiMF1l2SlZ61cReKFSM6J1UVw1ZDjkx7PQnaw9AMEaPRDUyLARaFMM1E/5EDTSUIYlHlAlmKDwTpELX2tLcggIi6+vOb0xgo9HV2lgcgJYlS07DhgUbFmz44YcNGzZs2LDhhx9+WNCwYcGGDUuNDA0NGxoW/LCYpYx3zz3vvvvuPe/eynw62MmMbdUQzDA8sYCBjGILfwVldFIfXxYLSPgdOKReQXjaQvBcgodkbwEp9RbStNTD8xQ0F9H6Wwgl8RSoVLSk3EMNyDBUoYYOnqik4J7tWoGVUiKP4alYAAAcJQzDcBgeHg7DYTgMDw/DYTgM/wfDMAzDcHjWLBYAAEcJw+FwODwMh2EYDsPh8HA4DMMwHIZZh2E4POyv/7Jo26raau1WY/O4+whJTnIB5Vevq22vJFlOXk6aY9Yl5GWUWZcSZl1CXEKZhbmSupS29/d97/9HZFZPihG9R8+0GRY1efADYeQBrQazMNAyykmL0j7gFVSNioOwDtCfF0jQaq8aoGqrgA20nx61uIIjgsIfbK/jN6lVHosigATSpDgiQWuE1XZZvAP1HbEdqtygPzTHJ5BAA0FdqBEWRZeZZtEbUQcbaWwKr4A0A2gEVRkhJwDbQJIEx+WVhiAYgiEIKQXk7l9QDGdKNgHbhjEkn4URCIVhGIYiJusZKDv7bwuyrbCtpc11Joolex1koQ064nl+f/z/o7eec+FTSRGv9axSL4a8x6fTMmKTnu+GPVJaPmCM0/qCaVQ+p81L8YjT+oofIF6KV4yWhpe+Kz6r9p/af2r/qf2n9p/af2r/qUlBfCFiFmKVRNgXEmIWYnVE+KHwffJDL2jOQshC/HxZf7WfWCokX9un5Af1RyxO4Ifa6ItCrJTMkyf0jWdzBKvkCVVNnmv6F3Zyvj6vG6fwvE8eKMQqygszOV6uwfmFWFH5+kxOl4MoaN+BEjyZgovCTBwF7YDKytfn1bQSgTcovZxEIbuu0VFWluCVSdMv4XYyXS/L0jJyPS+hAV3kig3SFbIHQgExhBDlG6wvpVo39cDZnrXaSBFTiGZdEDrfYNNHMTDLKd9gvagGnyWlWrF1BbUgSoyiG24GBhKEZLMuCN1vsMmjGOxKzE5uwZrN0ihL/q0bJZsD0PvHIwGSgJOJZl3SuN9gU0nAw/qZVMiCCKIDZt/N4nPIza9YS5jrftZsV2TV+gpbUhOSdbVoGmoi5sV180B1eWZGuh4mWl8htN8loQCcWSOzPOLEm0uSjtRpUt3GNSFACJDVg8X/i2GuVDLSMpzr9BnXCsWHBKrJMmaZIoIrgwWThAKw7CornU0mgStJDpPqNK4MEyBgVh8Wn/+5Wc61Y3O9Mi2teqDjRKyJYNK6IdiQQMIyndmVjGYWECEHvN4uUcXMbDialptKsupBzuxq8DCpxIawkLEN0r0y1mVKdXkyMwTdbbx+t2LB9iVbLVGxkKI9TSB3JgmB9ajtQIQMMJhZMzTcFc1j3WzSXF5J8jCpJISFjK11VV3GtlgAaL9Tr0xLWf+q05KG76So3aAd3hSSBRHyfgPYh5tM4icwO9KOuU0qoUEsZGyhe2WE1RVbx9ALdxtPb/CCVCzap+IOg+mNdGXm+xqmmY42nVSf/J5mnpl5mFTq0SOsgIztArnX3iiz4IL2N/xSeaIOvvjmsYDVwYCibeptNok7/Pr7fExqJEaEhc58A+FexaEhjqaUVDPdxtMPJBPd6q8kn5VnDun2jK8CBAmlspKGUDaV1LFgt1dk3hy4TCqPJiIspKYq3KtUxPqTTzwOMYA9UA9cPx57dBQQYSUW5J7b9NGBdsyCWVd2VZ80kmWqHkCTin4gLGRsGx7hXuVCMINe2OMbUUDsnCc6mZlDxZ1z1ZPSoAAhwFimVO81yLwebPqoN2Ljb+/mbKQ9S7r0O5N6gEyqVAwQCxlbh3vVf2y2l2fIC/sX3vgC3hgcybCBYTDHZWYmWFwpCgjDBTINN11ZusrLPzZ5xU0zaFLRD4iFjC1yr5K1HiAv7GP2I84SjOBWFrQd61AxDEgEshMRhgsk1LAfg+0t04le+xidwKornpy8qAVkUpEgFjK20L0GxrA9MwDdyWH8y74jxx/ciAq2E4oxChbX7QEIwwESZNAnrg1owW1StSkuMracJWPO341hDe9brvZHFHKA0lHYgtOkah1CYwvdq6e7shT4pcDmmaao3qdOk6pxqElsoXv1dUfU4r4U2ExSmRNbksDb3P1NJdmc2H6JKFu1OShk+NGJbdzqFRiua5CJbQxxvSL9VyYxYpAWh1iiB2DAIgaU0oq1FeJ6RY4LHu50mRYvkIToBmyCFTGwVBqRao2z9ST5t14RTldwJ7u2mN5gwQ3YbQIUzJC6BOe0iqt3l2YWAIgaxGDz6bNzrb/SqnPVwxLOlVVNps1kmWLTsnrMn7V9RZN3zGv2rfAEN6RcZmZwqqptUiOtK0VYkVRV//FEVbVjaK6saqLqZkH37nY8Zt8KUagpKpO/pzUStK70QKcz28tpuyOdlmiurERZaV3NXLOw6qslYXZcs2+FqY1GHWoVLlz+wj/oclp+9svQpK0NOagHhqdlBXNlcZbeCLPwKXrYPf8+s29NQxONmqIWCyA4VVWyV6E5uuWUGshthcBcWX4sYl44G+Ydn9m3QhNNFxpqFQhKVYGcli2gSVvJA5px1YfFV3/yRrACGmoVCEpVoZyWBCdtbaa0Ac246sPiWICzbwUrgKFWIbSpqlUTITBpK3kW0FhZGC+OVdDsW6GLBsmfvKEDTtrabUizAY2VhfNiPi/08eO/C1wbXWVDP7q/boEmbaUO0FhZkBef+bMaSt4xMPtWuAIaarWDdnR/mHoAJm3NQI7GsCEY2u5Sld4LJOBz3EOWTFmGqwf6+DHigT5+DNLlLwT+K4X06MoSY0DbdVXhvFhvtD0z2yHz5TMza6hqTvMzcHMaIFUF6T3UaQkmbW0I0hE1WQ9t16jCeQmwdfaamTVYNYbtIgZfvcTnud2GdMNIJPJunQMV0wbRJ1q1gM9zmynp2I8Btc6bYpoiRjRRLeDz3CnxQK325tYUjvQ6G0YbR9Pb31zUenSG1i9PQBHkRLOdX4gR/DTxI/gkL15OPUWbM1/wY+8eLSZY9VD4S9YOrdezSqV0Fyv7Y3vQ4F6BO4HZF6vWD4X4nIsVkla4BFqldF6GyU1SsxCrI63Yryz2T02WmoVIHIhRKlkKKw+yF2LtP7X/1P5T+0/tP7X/1P5T+89BvmhO2YfuBKxngjCzAjNLEESYWYV+zwThwT1yP3ItnGVw9i5LAER4sPrRm1h5o3vhSNcs5Kvek/TZQm+HKX257obPZCKt6Q49ngl3AE6HKU2ulmlMQGb9pGhL+A26PBNOWAhYvycqa8JDlmdCdk3AFbQOhcz6Ba+p8iK9pjwEeSaAaxKu1HU4ZNYvdk2UF+k16aHIM8Gva/vPM3vW76ytP8/swMKs79O1sP2a7K1LcJb4lQopv3iSWTtTxbcf6lmvr/Zj/9j+0NYsFQphB2jWP2GHSgou4/UueUGHLJUJ4dNEK73CUiyvBLcJVgn3icWyaFYpfcG0X6o65+tZpSsWKT19Qid9JpcdCDD7wcrF25d00p0COgSkuYv2+335BmtdXSopSziaeXCfBojlG6x1NTSSQjCaN7ikV+8s4dx4f8LZT298XVpdBml/rWtKIe3vAGdrSQREOy6IgAgQfiMlKFsLILjzuPoQARH48EZJ0he+vdxzoXpp743+ab+f3vi4/I7q2VXxwv3p3UtPu/d1CfebAVF6uTp+ofu6QLh299KKsyWUlyTVS1sHr0RJCqWSkCQYDGE30SWcC72EngKiRJclgXBfWbm6KymPX+iPeOMkL3wsSc9P0/QnqaEo9OqSrPG6XtN/91aC4SI6u/aWiHb/1K8//uY0fTkhYmx3r/bob4e8LFGSs8vbD964P01T7iFSevburcQMACJaINCU49y4/eDldy+t9Pnpn/r1uwmKizdKkt5XZT126e1Vn79weyvZvg+IaOFR2me8HC7Ryxdvp+x2bsTbMRsXYU1DeuN4hTcUQMe/hAkRId14LEl6ee83Xet++tcLfcKhP/owFnLtu9SP79P6vz1l5ca3164/R4bg7ZSXu+wHIkIL13oZ6Z9aksQCY0O8lWUjxogAWkBrc1WtTvXGuF5W0iywQbirdy+OH9Fa70/Td26dPb176+yqlsbwLnaZABFJL+F+M1vlPd09pXdlq1zTpumHL3ZTxFtViCGEKABaQ2tvTcVV+I2FFtd1MrupvPzVtC9eSHdT6e7PntIPB1FxdmnFXf+fWpKk/xeUZ08hb0UhBrMECQC0gtbWigqx/XXLQgtxHSfDhj98SSfRN/4E2fq7V3svVH/kr8q0GH3fN6dMgIin07ySEJ2m6Ts/RyjF2VOO8MY3Y15pERjIOBVjmKExkl9alMGWOEtumsZ2QFQHS5hFwpQADPMsgWsFrY2PJ+BfG1HtejsaW4K7LtYn+rKP7ovAOBAG6ZFESAGNFzoGAxmn1lgM3J8a8GlRBmtNllxUklwS5lnyQBJFtGuREc0lEIE4ATAHp3z3CwvIv+9BgNj3JWr1md4SPag+YyGui/UJeP+K960sEIOLw4AbXNp2K4uBjBMwJhmEPzXg06IM1posuagk2X7oiJ50jGarkAq88JaTKCBaiDG4ATw4bE4eAwJ6osqwZMjL8PL4L2q2npPF8Um1gRWHgYwTMiYPgulbmVchAz4tymCtyZKLSpLLwHBpPDDWWnMF44W3nAyI0CRN8ADw4CA5AD1RWVgI+skDnTZMuQGVknFqElEwgdQZQ2rwKarGZ54MdqrPYN1ADSKDBTxhQ7MmS0YkySWgOy3Ndt5SPWwmJNs5V62/r7R5Tc1NbzQtCa2aZF1dwswMYNl8+uxc66+06lz1sCSgXPVkZtVEVTMMWNx5oPrazEzAS0QKcplS/boZi+8KkzcI/XnjId6p76/aJFDZjFOXiMI5ps6YVDdHiqrxiclg9dkokMFCII1n0JLqCRva1JYsWZckG8LAgWDyMs9oe1XVezlA0MwOtJPoQMdGaNXkXq7TEmCRPkQEDTJRVe1IoGLBhQGLjxNBPZDwADGBEwDWaA6te9iOH6SQBufsEyToOC1wp+gwkNi1mlQrFhUHgvLndKMGHCzCkNZqZh/BuwJhy64sWU2SQXIVoQHlIAbW+agm+mppezlrBAiheqDj3ihLNC8WEq1jBrFsrtMZsU5ndqTTcq6HDChj1ewCrBuCEUl4gMg4qjcTtsAqvw0/iCEY9wipO6XPWE62UZyp6AMNULhrle6Ry44eM8MJ6CBmgBY2RwYbpC91O9gukMJiMlgghUWG1rUsS4aSZIV8B3HqafTKA0ZYp8OOlnhYC8nexIIdaMYRWIWIdTMjtETUNDqw6oExHPZhuJ6OqLlnHCQnoCDilD2AeN2llku0pyEy29SKPqfrlIbU0LdT+orvM+yGDpLqr+Oas1MVt438THWgY4Ri0LUWC66ustJBdAggQAubI4MFFvaO08F2oRRWn8GCweNC61qWJUNJcodvpf4QVj+ZVxb0NYmoj2O9Eagd1Wl5oGP+EehOS+HWMNZcbFsyNRNCpzDqAckNGBgRpxwCxJ/OOyZ6Q3zgY78lPU0k0X32HVylv2eBm1J+HrEj7Qg/WSwQCrOkwD0y9UbFzGyYkAE6hAksbL4M1oCFdTtYKIXVZ7BQ8NjQLMuSs58U1MUuaVUY6Ed1CzSHGbUDDM1DxaJtKhatyuEWWtNdNDTh+y7ji34OiRxnovrk93zGknLADtrMEAoHE+5Rfo6tGX7PiuaxTktBBzEDsrB5Mtiyhyys08FCKewb2gz2EyB4bGi2Zclqkpzh2Vyp3AEYanS10qmn00z/Z/Q1roWOydItmk7O22kpjAXkV80QEFYzyaOaefMptdH8s+jRd1V5f7g3ythTAaAwsKF0j1zDXPhNSYcAoIXNk8GWQ3SOczrYS+A97re0b3GPgeCxoV1aliWrSbL654KOVqy/G77hgM3BKtWk/srS7CjPqD5W6HpghCYFsVxymaR64ADU7RkzSQLeYZIaxGzPkYO7sdxMXtqy2yuybCeYiYQoz4BrjVK9kb70fQ1zm/oBABgAIAsLZ7Bt0598kgs1+EtsaJdWZskghZ5rAAD6miAq5aU8AzXJHRDtUGr4pQLCcgvydlUzDFg/xgM2kF0Q1AOk9dv1dtfsEynmOINRebK3TLEPWoMlp4hQ/hFZ0p5kYWLjTNZLhIABALKwyAzWbwHMhOGNPTQbQPDa0GzPku1qToO7hrdoO+c4nUZeoehqxziaUERYbvELEoefsWC9ZUqnJQa8c656gtomiLjzUHV7RgsbVHHHOcyzz5jtTMQY9tgCQvkj9OkF7lEoK22HPruA7qcBObKwOTJYvIrbwUIpbPYtDnIIBY8Nzb4s2WpgSCYuCvikDUROFQSNkzAIKLQdJHdwsP0B+WJ0I9Jjw6quOK7AZa2ukmVGKAwMuEexII04AAYA0MLmymARkdvBQikswjsQPDY0y7LksrCPhg19kv4EVRQ0TsIooNB2kBxzcEE+yC4IkscGVeCZ6jty1XsNQghDdsoEKNy1AvfIDkQGce8fz+sBAJYADgubK4MFvtTpYLtACovJYNXgsaF17cuSywG4xu2T7A+oBI2TUEKh7SA55hBClI7QRIwbVgWnS0x0oGOAAiyjwOCghpgQMAJhrZA+HsYMyRGwix9KYWHvOgvY0BrrsmQ7KeUYN0Ct4W9cFV0uMZjtsPNolChOLxoj2w4hed0UIqGbfBksAnbxQyksnMHqLGBDm9qWJTsc/3GFjY5EBOgWZc6oGMDlEoP1RqodC8guQi/q9p+YAPhMl4XV2JI0Ls59wY9IYeEMVmcBG5ptWbLL8R5XWP/Kir6myj9gIgbAoAGcH51qoofg439zW9hShObR8dWpmdzoSPhfM4cmE/6Jf7n+FSNBiDLOY358/CN63NQWthShefbwCRsl0YPNo7xWNW00n3fLnMKWI7QQCXg4p2n0+uYQRTHVfN4tcQpbjtCCJOl74uGcZDn1maQZDSEOnxAkSR7H35w80Nd8EN8P4kb31vMb15/Ts59N+wgpxiIf6Gs+iO8H8fW3nN8FIv1SjOXHVfkLoq/5kEjfTmm73dP0vu6mCCnCom/cKEsSzaMOv+aD+CIQ8ltCPD9FSBGW56d8Pfw1H/h/ghhtB5Ei7J/TPhl/qONHbkGkOMvdW/rb8dd8cAsiURIanIkyoVvL4a8H4SGEpEDEM1GmwXmkk7mt+3t6/Ah8zQcPpQBJoVqcR7rrRWIIoNBd8DUffASQFKrELBDuU3jTdt0nfMDR+sDuQkXfjqgSACXDem4H2mjz8VNvek2WPYuf2oAXRN5hJCQAvJXFczvQRj5opeLp+WLnV6pBoL5NLI6Lnx9dz3pdpHRmf2yzSsX5Kyuf6SKloV1xNF5X/YgStl+rlM7tnwJixZrcpN/US2nlmGeU/ceyylKCCZxWKFq/HVQui1efUcLvptQrwbs6YSB2iat4torX/nOAMeOU5lGbXu0/Bxuzitqc1/6zz5Ob2M1F1OahcNymFa79J3ZxFbsZxm12sZtl7T+Ri+PYzXJfN8PafyIXm9hN2tfNVezmou6bAUvc5nw/N/fIxW5Wdd8swi755/ONnTd4Lt/l8NVa70tDj/PvVnr4qhIIIyU4XxkUXzl8l8SXtNb70pB/txLEl4CvOvRGU+TCi+0Qoe2aRS3kHz7nQ8jPSUrLjK9zjS+ErmaQr6MhyteLIef1dTGAfM1Rvk7B5/0rKF83uX29AvqaoHxds4CcIHzNIV+DC5TvMeRrdoXydZTX1+Ft1he+Whyu8/raLvJWi+157moxK7paXIDFykJvwCLPtBdBaaKA93UNsc3t6wTna2zS1w7na53bl/qyJTxKhfjaQbyS29cY5+vEpK9rnK8t6Ct/tcjta1vOarMGinU8ZyE6woMRC4EWc6ot1L2S0t4+JjnCg11/zUKfP8+J9md4AkvkRlfk9LnbnGh3qxBQ5f/bNYwJVrsDK/MDzDmK3fT2iyEQ4s3SFrhVQuhGGGpOw64vbvO7+jq7q8M2wguoMez5wI/lLTyCdvV6wBK4ud5Memcp3SPfe/e409HKj0yG7/aOUpr03v12p+9Y4GawgfohepNZH+r5GrBB9Hj24RoYQwLu+B28GWy8sS5kV68DLkZX3lgXtasDOIMNtPbtahzCaYWvgXdQn67ItgYHgapQDDYppbFnGSyJtoZG2LsZkJ3BcrfQInnF15l3UL+Kx7l2tX1VwRfS1W4m/AitzPpfzqvOasze1UmeXS3WVQRfSPcjhB9fXUT/e34xzKvlbuhffbuPNrQ9Y7Hsd3019IFuRvrScBf+SGxNFug2EUsjFrs4Tn7Qjc0QJUkzWDbCS2pxbhtLT3DhSoSPVp7QgsUkm0s66YmVPLp0vn7OnQiPvfLmZpQulWQrzi91t7LyBz3TNDSStbi+1N3LX73pAX3R2CKYUkhWYzZ8qDairuWdgUPEafzcpQd0r8Y8iFQ+YkAltg813lwSZnEaHvgEjEBX5BWPGAyVENuGGuLNJOF3Vywuw/2fgHMhPJqxVDpisAPl5d73lQlxm1CfWIhrX/quLG7D7bUfAPnbT2uWiohqZgH2/iWBBEFnmXe/sMCEHgT7bvjVVi+lObHRFXkFZGxU9vJ6YKj3HyWBeCXEYfMlDCkgIDeaGz5u+SmEIABA7x8QiFdCHCTIEcxJ4AMf4pahbOdctabev82nz861/kqrzlUPSyLKVU9mVk1UNUOENi/uPFB9bWYJUtbVYgYQg+0sU6pflwDErcKFZpPz3j/ThwihQSaqqh1JVCy4ACHp40RQDxLdy3VaAkQBx/oQW3mS5dBsrq+WVj3QMSFMZ7aX03ZHOi3nesiIMraACedaNwQnqibaMTOAyDiqN3W8Jah7DWI2zDMzZu4T1QMiKBa03fS3jKalBcZBgoQEKIg45fUWIP7JUWa0zhbgws6k/koL0umrTkubi21LpmaSd9gQXbcekDAhG+FjRJxyCBB/OiEQG26oZX9LbJLdhPt5pGLRPhWLlhU33ipBS2yS3c16I7Zmx0RB8luAhHCBhNgIG++FMrTEJhjsDaZlNam/sjQ7yjOnXKvUA4OEc92e8VWYupqbNIjZXsIW4rpA0vm2xwJ7A7bsoOOLg0rREtvJFDoWc7+/BpQAzNFlt3rgIUjUVc0g4bx+zPsRQqClUc4B1gmSvaPYJgsMAJFJdPQ3zC8KcldLbHdQblK7KfacIQi5v/oSf9o5V54J4xZPnzn8jCWAy5ROS0nI0mnOVU9KE+qCFnceqm7P1gkLMXDnG0O7LDAA1DtxYjuwQnwxkLNaYkd3UB6k1pK91HuFpbiBBP8u/bKKN6Iy0Bh88lgDZ/Egghyi/YF8TePhxrDGP34N83rcDtoYRDPdaJ0IGn/DvMoSwSNiDle1xEYYktCH1FYyl4bpzBDY/oL/mP2yatNIcM7o0+ZGzuJBhDm4IN90inrhXk+IPQYW3LYWOFNJAgR40QBAo69hXgWpqSsGIDM1haNaYkeEIYOQpFC4jZ1kL6X+Q2Eu8OhK/02V/bJak0AJTJ8eE2fxIYIc0qZByqneYtxfT4g/HLYWFGerjNXATswM8QJK+QmQFrf9AlJXHMI35516bWbDTS2xYwAYKAgX6TdtJXsppSEXqTNV/otCmb6X5Go51HWF7IA/PGwtcKYACUr1ggTpAAAJWtx2S01dc5Au9i6YuYmTWmLHYADDSQYH0W1sJHsppcNekfrVqvwXlXL98M/wukkHeUE2NQBbK6xp0yQZS8sU2eCAkKSgjBYtILpKEEvDXD3ImUsOsc1Cpa4KWQvrlDoXjOCkltiJehwDk90hUjAIOzlOtkj8sstSsp8Q4tqfBMofyAe0tdyaPpsoKx0DSEBgRisF6SpJLDRXcBG0rcKmrrcdFO1S5oKJDRe1xGbLAgOS/TDegSllPoDZxzOZtIy+aTw65Iyirf10gekfky1DOM7fZWk9wQpHVcck6ES5Na0mmpXW1cwQkhCc0UpBOkxcLPhJ0Y6002ZdY1PXr+1puTlTPSyVmQUZuIljWmK7FFvTn3xiJYvrTZY7AKsi9S/Fb/dnhjciG/txgERBE2Kbf/jHtIYmYPYFut10R9Qvh05UWF9kaRGSEJzRSkFwRCxUn/yez5i1d8H4W1fu9r9Oaoktd11wLYCGCHhQK2/yUDh74bDQm/wr8Vu7V4nBWZcxgCGldsOH6B/TmvnPJ3iDaWkH6nCi0gSg7dxIXTCjBYLgzwS3kHVV5fWltume9cm0/jMvTmqJnQhgYLK/fOsSDMJWOHthw1LgA/24+K1vlhjkacmRj8Gy4XTczk/lkO0Q/5jWhNrWH4wtkH13ONEWVCxcapAUd1wKdvsc9GANLk5NpK4bBQMbbmqJHQPAQGRuUlsRyV44Yilwmlv8t35b5JWv4czHsDmvY890D3wFnZikMBeM+9J7QwCqiY6xE3XLjdQFM1ogCF65xMqTvWWqHrRXPROpa4blfGkAR7XEjgADkElSKNzG1saw2QuTAvkd7LeEuj3yyNfA+Rio++mf7oGTNQhCCnI1HUPKBuAEYL6H7ad6gJwoAjdSwI4WAHKHxc6J2WfMdiZtV/7Ulc+zLTps7IgaIgH45LGKTy7CkcYH8EoK1HOTjWExTu3typK5UGD9F85eMgaoR2OC+wg21v7pHmBfgQC48L4Kzn3rRPE97A2KBXCiCAwgebxQ/RjxQPUAqstfCCR/tTedJm/qKvxKpj2mmWEknNUSG/EGh3Bqd0fUwk7OCfJsjhhAgrD4MB89lokZqpodsA+lSAhGOcTO7GO2e2znLdXDe3mGsjvEEOr1dsSHmhwZxnKkyffQftcM2FoMAEkKOFqcTgO4Q+KGufqOnKWnhNjubDkldcVzmGlsbQZ3tcTW+mn0Tt02jIRxPg7POF1lpePQMJfXIEVGsCv7GJ/L7uVojCqzA+0YO4Lwya1YADA/UUQGDF0Z7n4OnyuZOS02tQ0OKs7b9ltZ8qaukv3UaQZ3t8T2ByhBmHS9HRUzs+FoWuKFuR6WMIfYI/vYQiCiV0tiyWB2B7s0TAGQUIaxI6PYOCLiOsxnwNjmNsj7QQyHo0V3xY4RUK3irSzIEoZ8iJRthNY2N9XkoB7ghAJrht+zonmsDlHOAc4hDq7sYyb2JswhZDD7WOR08PEomQeCM4oLIPN1iOMoJANGThR5P4ThcrSYgVQwMb6jdRWmkcDmCh0/JR2Aub1NNTEQ4yAyHVDYdZeGKP1QdDHnclusjoXAmg6YA8Ju2NAxk0ALwZlRbBbh1eShMNGHcGyFZ+csZaWtTTVlYAZBX/q+hrk9Wisil5HRVlARK/MZrTodV3zWT7dOPHwCbTbYzULWG/kvuOW90DFZusUnEwq4EMyZUVzEEI5kp5fSvIKBEoQrpqyk6z1M9cBjFX8RP1gF5ykP8xVa4qM8EJRRHMFACcIVWwBuneVjOHM6WpKAeynPDHESlgI6rwzjCAZKEK4mU7aG3msQtn7vobx1CiQEt6IYEjhHOo2ezNhYgnYscF6QYezIKI5gLHjvG6xnjAAjOABCDIwBk7AIzCfXlSPG0EoybBwDpWIE5xT0njkdkTFAEh5C8LpTJTIQwet1K0ssA6RioAwMjOAASHAhA8jW8MvuEIiA1wMsnnGz7oz4xb7olO9vVUv9UB5SCEonml1NCEuniV9JCEyTvEBD1zNRLoL+Ixfpswp2v+2kvbGxkmdS8cj5+hYSwp5HunGOE7E0s41vJz/o1K24X6a5PRw6SZuUhm7Saq/W28XR0AvaLlzKD554QHfR5EbnxVG6SKnsE1Oj1+QmnYknZJTWUbODzVD2aSVSa2qyrmTd9wWbF0POhwOnUJvBAfungs4bXNgjkhbHbQCjFR53qrabaMUjQhff4fEseNFK54uRYiFeke8VW5jCquOKE6/IpQ1ftC3hadu29p+XeSMilH+gYIVVxxWnXkum4IXw5GLvuOI0R3ixZglcCN+mdEZi0McV77GEKCzriEq9Iq/9p/af2n9q/3lZBucp3ZKd25TOQhet8FGnalN9CMeL7YEXxxXXnLJPhD74kTBzlkyKE50Rnt0sOAOP5mcsEXqxTXcArk5Smgz7/VmUZp1UTar6RvVdKfRnkZo1tI7SxxUX3gLrGHNccYeuJpXt9+KK8McV740d+bNV1vHluOJ5zhLYMeeZvc6sHXamsFbVfE+cOG0F6ogqwo7RtrNmZvbeyeKLL057uvIgzD3H6Ptp23OZLPq4Irzo+UAjloqD8GCZaCVrGukK3yQvqH/EUnXoJWJxYQ9LXxRDJWK7opMsY/PE+fpBVWLOdFIGSsx+sDohVFIWOsxdtEJBpgJVIMPMg+MIL+7/R3iD8kbwhckjCbzwami5/ub/+027wzrZnQwt1yssYZdFIo9GLEGXXkq//chqqe5sD6tvd1i7lM7DL/95abVU3760Wu+kzaXVejsMY3XjDpLVYXWwOqxQjM2NO5hsDiuDzWEFYyyu5FwWh5XF4rBIxayf1mTnNqWzUAW4LxeLvVIpFOLNjmfkdSIvboaISmlx1AaupmQ9WGmtV6o3DqpwvxpHB6WxaYJKcVxxfxMDLxEzCASJVFKiozYirKYYJUb0rxZMibaDKxxDOONFhAHFa4AqcVxxfxOD8RIiZuAIAKmcBFAbKHYOEQLHAEgtVmposHZwhUMIj3gRYUDxGqVtCU/btv4mBtt7S1VfawgLmCHYgfKyDH/fW3vGy0iw+WtX1JLVRnUleatj4kRuzSbPzJ7Io7uaTL2rBVF2G4Sdg2JhuMIlxMwvXkQYULzBkxjsgfJyaJgBSfWb5UV1zHSgyhSj0LtfWFiISKLD7ui6A0qApbpKNM/MUcnVREB0zBUvRNNg0UTE3yCUgEbUwCfWBbUGxA6U+mNm1kyWKR07GBKNjcpe/vHzli67TTmpH2dM82U4T8T65VzzwiE0NuHsujMhLEFFtB1XJc91u2nMqiOqY0e8Cvow2AKP1BGRiKzC4Tg/cAp2qvI4d3gCxbYqRtfhgEaR0JjSmlBXxyYOGMzAxI4gAigtxRWZmg+73o7+IBM7lv2EOuzurjsTgJEANvyegaOSD3RaMqwDzQzHCwCGkRGKFEcEQqxw4PNDZBSRUDzOHZ5AIbZRCao5Dge+XhS8gmIt6EDHMjTMEF4k9Ml8TDVQfJIUbOct1cNmkq8gE9FTkbKd81z1ZGbBjh6r1t9XQsqMsDhMQpl1tZhZd1o6KnxeD0QYV5wPGI5XpYHCyHr/QBIvo7XeaFqC0GRk1Q10XuLnDMcpAzN5AlmIrQuM7/Z5zkfU2MGDR5HQmNIaUDXRk4+VVDkOhvDisFD8yVEn0XzaTEg2zNkQo4cEkBYL6/KeO6bMJFZGupfrtKRnUCWjfcg+Ro54FRowjKz3Z4kOtGN8/4DQZGQK1eG44uagD/KDnA3t8g9Vb5QZQLH59Nm51l9p1bnqITqbsXahmTwkohBbFhif8lIPsMSYEaJY/VWqc+UD1wYZHKv8XnlvUizm09J6o84wkVUTfbW0vVzFQghBaK6HJe2nrDcqZmbD0bR0UOqrDGtMMB2zJJKxo5J7o0yag+CON/tyA4aR9V4lYrisDlBoIjKFCnFccZPMxbHeG2XinAFQEn2ICBpkwk5d8GxWTcDpLLZNc12G/3Fp9uRBws2tr5nJW8S4GEZcbQWzqpGuaB6rTkvDDED3/nHn/9tLy0E9sOTvTMM8Y82dEBV5eLM6qyZE0wy/5zOPdVpCymGeccKMowX2DK5wIfbPES8AGEYnPC4Kn59XUGg8sq4e4cPjvRaZNEd4PGcxRJRdvCPtiNNFscAoOp3ZXk7bHem0BGczUlZaV7M2iX+OtxfGPvxdHQPJzgUXR4gFb8jD803NHAy8wdyZcKdTXnqjjs2nJRd/id6IiTd7HRI/+IN16wFvyBNBSmIRNHwAkD2DK5njxH0+DgjjBYDCyJgUOtCxY1l5ZwGiIhxX/DxzXHGD1Ce/p5knBsDYOQOi0CCPgdPKb0FnM87SYyDRDGJAOVKkK8oPM9F6oHPBxUxwaArB0UpyF1L8SciwYCN5hKCdUi/Y3utY+/SPRvrS9zXM7ZGnpPtir+qo8IVcJVhCAOPNTTVhFFDFHaYq1hEVHu/KB2d7o4wdexBlLv0i0z10NpOfxXYpBokoLu0Cgc4FU29UD1b9B28J4dFlDgUTsZUW69Y//T0Da99Cw6Nj1qV3LnQMUspncCUPc94YGAHAeFEbMNblY/qGQqs2eKJP2O0VWfjObrFgY0sIZS2Jd0PpOcMCnQvSswl/FMwBv1S1M9EMMciFwMAyO6u0DPO/yLbrWMKBBunqgecqB7RXdh7qtDREWU3qryzNjvJM6IBewlXJc93+mFn1HbmODcWrMm2AoHUM8xWavQMKbaMncaA/2XtY0znlf9HBD1GA3AurJXol3pOUQp2LRB+ixxrYrKu8bC8MMkixA+onywy1o8YlrPdLeWZd9wuBYVZp/w1RIiwu9kxwVDhOhoXxQkBBgzBcFgMKbWMn4Q4/w4a6Buza6tgQilN4ldVa8BHiZwdCZzUFXdBbubhhBTNIgT+VFZvrWCjYzrlIkOkxYxCYQDLLIe38h3SJYV4PDFIyAj2ZGRN7JkRnhVdHf1FVX2saHC/EtFHD0EBNE48BhLYB/DEH93dYV2kZojgFTldAbVxF+F7+rK/qf8SfWTM3olqImCGw7QRXGRcycr6eu1ZM1EUM3mlLAtu8sHiFtlLhHvHCxaOLIHrEIHgqHdV35PwiNB/nCRDFLXA2WxUNcyINoQWxdK1sDWgfJepiBoYQAVdT1mEk5HruWkHTP7iTjl0VF7ywWIW2VOH7zng1xaOLYN8jBl4P1Q18fgjSN0MUtp7rBGUulvam09TfRw3S3kP1VzVhgdsyCFTEDBwB3RNX3kGgRPjuWkGzVrlvGXJVXHRiSegWK9wjXtClLoJ9jxh4ZJUMj5Os2Y4wAY6zUoyuE1Rws+z7S2/ogcoxp1nlK24tQlM5hnCsbMDvD3y0rUNA8KwETyjgHOJkMdlTjWfK1ycfMPOXzbWzVSRExakmergKH36z/cydN6IiDAfB6i442rsYW6UL6+xH0YFf07PEOKeRQBgOglVdJSYROQKJLdKFdfb5oXbzTQLFTvIugv3VUwx4rJG3kq3RNUEdbpRf3KdwjMGeyCuFvYeqPOfVny5jKuzitxFacf/1FGieU6G5CiE8TKeAyFriIaiGI4nmxULeKcMJQvTAQ3TdBvGdQzQE1+wzGNa05J1/DzxD6mL+xyASAg8xl/lffHHThapmPKdWVVcQ9tTOQ9Wvm1kAU50COuSIdRERtZul2mWyWHhJQBXoltQOEnLEujB6XPF1Sme+Axp9Ahpygdl9Vx4em/GpelPHhqY6BXSQYygMAgH+dIqH6kM43pw+FPYd6ugTmtFMfcQmneUDAmKqUyCNYygMzxMMjyuufmenHc3UJQnFx+lFD0BI71gNw/9UquOKE4JIj56Y6rRtIsiZKFePgx/5EjMbbPFATHXqu4CCGueRHvDQ3KQeeKzSIGZ7eWZoqlOi09+kHLhe4Q5wuNLMOKcAAkowoalOGZ3OcSl4+qbzdZ9CsAMYd657OdAMLwXSabZnFtBUp5xO47gU+EBBjS44GEVXN4SpdoAKxOQLpQ/AVrae4Dqg0e12EUNGaAc85VCIyRMKEYCtvBhyzwu61ZSCiEjAOuEucDCKqW4IU+0AFYjJF0obgKW04o3ZMeh3pYSq0+WzOCxracVHMw9+KHz87RkLzRGeXw9YSAhBKn9FOK74IqVxqKJCDCNxMCGhd5IXARmL+zQx2R3WO2ljc1j0IfxOsCoQcxfNE6ul+tTusPrpHbvDehp+uU3EUfiFv00bslWeLoTiiUUHXuijNuxCoQKt/efgC0b9tKD8IFBndIQyP50RHKk+hOMrNwO7gLN0thHVqt3Y73uA+NcOfakUxxW3CMFzbLcPVSzbiW2qVWYW8lKFjiteyIv9e8rLj3zTabuohb8RN3vqTT+9XB0/SvsY5OWl1W7a9+H75V/8nw9YqEuFOPiRnajutknCP/0ptB+yxFR+O3v2wxdLOE9T2PuXBBwEcICmQCBDemta86IuEBeg5OJoDEQIAnIeZUO/UwBA7KDf7wsA0PsHBOKVEIdoClIA5CTwgJeVAbgQJUmiMRAmDNj/BAwMVAoi1FhC+s4rqmfU+0+L51//ip5d1aZ3E5LLyykRXai+75vTu7dUdRcRpsXxL3ld9Ru/OU2Qdq/e0eNvBoj99J2fU/37CQAP8e7rFwTIHT7bDkH3E8g/o/qf/t0FjKkgI5P+qSXp+SlDBWEAQQOhSCTZePmC9/6ZLj8Rwl+9par6tiQ6fsQFCEm/jQjOnib6kQt9fgoQEzgB0GcqlAMiQWgB+SMXMKb3koB5zZBAGHDQSbEtEeQm/4mO0l84Te++rr+dEJ5/c/rhC9ruJ/X5aaH/iRHtMiOPCQs9+wlOdPeWvp2mKUBkHHev9uhvJ8/Pz/5pEsTbWHp84y2A/oVTCnQXxfSnfj1r4I4fMYEwIGNAKNSk8zVK9Xygv8o73Ckz94nOnhLB8SPWALxw0PHbZxwkSEjOXRBxyj8FEF8o1CrwLjnX2fs++P6EFglCs0ioGnZRTCxEWpXpT8kwIGNqKNTcyNYVqiOdIrdv451bZz+b9qXZVmV2/RFsAP7dBZ2Y++n9s6ckTChG+O5Ls/8yQLz2mhDgmZfuq7KuOlxA0Ay3T40IjCmpFKoYLk4mmt2jhkLLhfDg25sZC+kbw9pGn/t5pONHrQoTtizkcvrpp19hjcHXt03kQ3apDYL6BMwffaiGsvcTinuqLVI6sxDm2J+yFkEUJL8FSAgXSB4LwOX0GeVf/PDPnX236Dx4L+Do7x/faKkxQQtEDxkDQvnrcEwvpbmFsDd4fno3eWkaIrvYdQqvwpoCSFjoG9/MR+WY+o5V/mqafpjnv/Wpq/5+1v94Wujb5FOen0LotJCr4Ohfvrisc/ZUSIYBGlNDiR4QQIEuu5099RAkuq+6CwmLs0srbtCFEJIgYMc8jwxcaFPVN36Ov5CAFpB/9ALHxBsfIRAGHDTUjjdyIP70zivKM2Hc4hks/+n9aQL4c/r8FBBSRswrCcFpKtRHiCz/5Q2UTvNrL9iVgX76Ky/0ffSSElrginQaFH0/vU8X24RAGEDQUChtpOATqEvuk8fa5yweRJBDtD+Cj8fgmwzr5kH0EFcEAGEEUxuSYaMFaEDd5y4UzuJDhDm4+pDyE63fysIJED3A9btBBTJ9oqVbWcI55dsRQjf93ttHyGoNGmyhqqAvDaI2bdsenAyRFHAhk0ItNz06KcxCKoVYZn1i8YAxsNIKL3q0EmXqvvBg1z+n/CBQRyxIS8SS0KZyD0Z75UL1IRzH8wG23IRYIk7l7rzq0n4A5uClvVddstN23vFCmAqHeKr5JnzeLXr+Mkv4fSD6tJYLv81pgmRb91b45nTAQkTCZE+1dUpnUZteSvN932A7VObucCk089Xa6ct11aYCgB0OQfIOlyBMM1/S2ujLfdWG+sDarVgAS6cJoc1zLqq+hAcXKF/vgr62CaHlGPI1w/nqgb6GCaFv30K+FihfdwDWkK/bk4TQ8QL0tUwIHc4gX0coX9sR5Gu8SQhdDSBfc5SvC7BYX9khbPVPQV+rPsLX7gb0dY8cytd17mrzbu5qc5a/2uSuFj1staA8t0mnEQvwXT5Kuz2M+RrjfJ3k9rWFuM7ta4LzNTHpa4vzdZ3b1xjiJLevHc7X2pgv4VHCaQT5Kme1OMlfLfC+zugN8OwbwnqFBXSEkeZtB+VrntvXEfi2c4Xy1cvr6xBebFG+4MUhxtd2kdfXFbwYonyN8vq6gBcoX5oFytckt69V7mpxDfsqpFqcFV1tDuHFVlsehG8tF6j+rN8hlG/rLmHqbtodEr37mdf+U/tP7T+1/9T+U/vPwTFEaGmI0oI46cVp3SjEa2XZxmwWe3x0G681+yh2q/af/V/ozVyW+0P5f7/1Ji3vRDP+xfW+m7Q8imgEVsz23lLV5ZmGMAs3Q4lrxBiDVQ+Ul5MyIVnzJbbxxb9forluf6w021umNEP/00ynjWoB8sIKJEIEEW3nwo0YcxrdiMGaTT4t6XWsmujYRGFIEgGx8AciQgQBbAcofQKIW3Ka1009C6KByWy+hLnzQPW1GTURO+eq9feVVk2mzeSh1l8p/rF+RM7giC1XPZkZwZV8u2qSdbWY2c5D1a9zBBC23hYCbw5C3BcI8+Ljqqr1wGyYK5WM7AQrHQui3z5XDifZioVDy3Cu03KYcyIUwNacrrejjLsBai2KxVzrj1n1QDOzub5aEsI4UVZaVw4JEOu0NDvSjs31sGSYDmnHjG1XvaljGICFLUDVAyP/UCyoi2CWIGREYbwtYSVRffJ7PmOcRWAiScxMrIMCsLAVuLCwEKxbD4a58jJ9BloMO+Dv1GXL31cO804CzzCROAEhcY+BAigtbNWtEuUqXirueMhun7N2opm0rOgQhbRV90JBvhBe6BgvFRIsT/Ye1v8ILcieA5dYYIsoAAtbduk0JtJpCGImx+jqryzNjvIMCdj87DNmO5N6IFbhxB2rJipECA1itpfjALbi5EqGTVQ/dl5HqwdY+IU0Q2zG+wvbD4WCQKoHMICtOO1Hx60slE5zrnpSyiwYPZkZkiStviNXXYYbxESmzOFnCPPdXE+eTZg49kPV7RkOIMQtOO3HgDJaSQvPZNgQOI1PMqwLE3FsESbDUvobuh+FixgwDrxXxQXKCRyYHhwuzC02edwFzrV1fJ+5T6GFreJZYm7i8mVDNSyzrj6K3ar9Z38nHwE=) **Figure : Remoteproc support** ### Remoteproc recovery and subsystem restart The remoteproc subdevices manage communication contexts for remote processor subsystem crash and recovery scenarios. Entities, such as communication contexts, must determine when a remote processor is started or shut down. Each subdevice has two operations: - `probe()` called after the remoteproc `start()` - `remove()` called before `stop()` When a fatal error occurs on a remote processor, the remoteproc driver handles it and invokes `rproc_report_crash()`, which triggers recovery of the faulty remote processor. The recovery handler does the following: ![../_images/Remoteproc-recovery.png](data:image/png;base64,UklGRqYNAABXRUJQVlA4TJkNAAAv9MMkAO8FN7Zt1c0WsyJ1wFCW2iO7C4cceZQ6Yr5+lwxNwK1tW7WyvnvmELp1Q6NkVEHoEpM6nLHvETiybauK9hPcISR1mAtzbR+JS0bYru/9+u+e+U9glqgkspYUJq/SiVQSVTMJRCqTZUxOUqgEAusGaTUoVJIVqAQKOpmqhfSKpNIxKsl0rBVRnnmR4ZkXmZ67ifLcTYTjbjwdDxPueBgPRBh3J8J4OJFijkjxhkgxR6ZYQ5ZYQ5aoJAvWhEKRqMQz0/gi/iIy88w0vnzqqMS34fJfZSbVzqOorD1u+6/1AnAag6g9bqt3B3oLrz1ua/SfcQQOMQUcgLctXsELYAkWfNPVHLZVBi+s503AGMATDnEEHGAMYIE3HEG154gl/5GmqCKWiCUfP3Pfn5XWexMsTAE8gUe3gLslWDifvmv9vbrv/BvVEdziCnCNIywBh9heAvZgaw1giesDXnADeAR3OANc4JTK5BQcYQw4xNf+L7FELPn87ODkDVzhDdwsbR5PwTW5/SeWiCViCW2mWEOt29LoDd19/IC99uCEZxaVVbpFvR+1x23Pp1vwcG2b27bZto0M927GvRe5d5ty7+m99yLaV+9meu8J0xPKuqiLMkVZTOPVK5n+4zJznsdAM4MZEPRlRfSfFiPJYtuoBxHIgA6Gy32P5EZvL/fb09XXnLxp9ZrW4pxhzeqWkbva6zqTj6sVFqZcV+bKmRkqQVKveVYkdq40rU29hjUnrbHjaqRDecPa0WtYa3OmNcktNo5fe/vM6qCfCZJWmKfX9Hmdcn2uXG78jdQlgqEjeaRpMYgSrrkLDWvj+HXcFmfQZBlygqScDYKkmXEIjPpscTyyptnGJ0vD2tlnWGfSIoc3mtbePuM62kEQGB23xeGFJrRho3HttjAWE8/OyD+aslAjw9mxNfugL/K1FsaVRcKi2VkIgdFEPay4nI/1GTynQ8PhH1ltYZilBDalBPJBJD3cW4+GhpNW2JG5E6JhdhkfM1IC87J2ZqcEFk74z8RnLOrOk7PC/gAx3wobA0T2voJ9wn+uRMPq7jy9l4tx2k4rykyBQ50ZV6c49qQEjgWSoxnCurYOmkA9UGcGXJV9sLMzXcx6OR9LuvOsmvCfK6ox0wrbUwKrwsOpzH0x66R0wLHsfQX7hP9ciYZt3Xl2mKP46EUhBl5tSJp/HXMCegIu/v802WFXNoGV4Rh3bNkZj56gWBVEg40oKrcdgZyAT7/tXqwONsbN+ZGKdYrVSvrnDMfKcHRmbJXbvqAgBi9F0R+qop5+SXQrypTA1RzTwzFMlFujbDiRSbUkIzwsxIVLxEupUBMXRtF4KYv2JeLrVumrWSv111qvRdLb+w0JcMxOgBlE/pF3dG5Rvzqz7IVllrE1HPsvCtF6jY+dVk25cX88uvxRYuOBomIk1RpVfFAjFjgDNTHYaDJfEmWr9RPlQ40k4JjBDSL/wDs6twI1kHzUcecuymQidwE6vYAsYRSr7TGeKQSYE8CjPve4t19XK2RWI9OONiHRtJ6iFrNa/rA0HMuCqM7GzoVhRQLGo3nMlc4+Fq0PfrogiiIyzlRsX5cOxGBDIiqN4ieiAqJeUj/Xkr7O1ioqaK9H/TR8chRBx+oEkEHkn3rHDqW5Uem95AV4MjVrdZJsdZQlCvWL3D5gFqCcQB5j5B5N0wvDGG1CorRLOGxAqck0xiOYoWenj4WoPF4dbHgPK8MRZB0bO/QVIQnj0V2i33+k8v51Comn1BAZYiRvhkhU6R2W3BvZBVAOiTeajdCxOgFoEPkn3rFDac790tlmc1RkMgg+JQxp5l35dEkuP8oUapnmhC5PSO5pUqr1M/VSitEkJJ7WfYFoj9mhICRFH2JnOEo9ce7DmiC0xxgJGI+uwuJ19m+ixKmTxKH3VfhaxUJXEFQ0IMzbkAQ4pkCD0H+xih0q6gmolxujPcaiWKYhx9deonIlgplC4DmBPcbJvRLxpUGXkHBaTzAwnMWPRckWdU/u2BmOEX1ipAw2GG6PR8dBEQs6uomgpMYNCo+CvAXiqDPqBOxRf3noECxWNeiuGbzCfiA6PyJhxRAOqG3xUhqoknCGyKOJMknLKedwvYSZL9Onrwqox+lrL8nAznCUmffBy7/63Yhf2GgNdm+jd7d4O/QpCoDHFybO4xMyCP0TtDv+gViQ93odrddl+IPJl0NCwoshDFDbAqU0UCWJAS2n/BnXS1hQhwQNOjKvr70kAlvDsU6yS4fb49FVeIq8r0rd/TU1lbZGtYe8rTmExqb1syj6g4wQRl8oaY1Cg8A/9R6jUNKUV9YryKj9RUZTu0NcwEwZx0683GPXh0xh50diAKd1f/w2qppQUKqwUhqskjDUYxkwWdLXS8iVkBmo7rGtUViP09ZeEvFhZTiqEDXUQ9BgA9AaTcR4dBX1xAzuHfo/aKBGqkkQcO8sRXp0N6ESMIj9F6tahwV2o/ELzdp96qlAd4jKvzrMFGIZ5ATME33uqacTZYs4f0050KKZ1v1A690Gn0fVPZGAbpOglMasNWuxaI/FqpdIpz880hpVWaqtx+HTTAJ2hiP7WyWMEKUkjEcbmPxyc+v9SyrCw6p03kRfCqoUamqm1UB8DbwexYE6BifADCL/2Dv+8kPr9cgtPvvcOsWqmmDxtf8RNm5AppDxBXICeoyRe1GBPXwDBzFA03riUSfeX3ufxw1GEr4QN0mEJfyFKRaDDX1ekuMK7d+O1FWlApinRvH7WhJ+29JwJKWUn8kKP4GPLffHox2sihIRJcG/Yx//uudzy0RldatH155HrixKIKWUQE5AjzFyT+rssLjwG3JGj8ncerwaAzStG2Ka40QFMdgYF7yUZhzieagiXTZrdVCPSxxLImeUmPGYqnhAiHs+9+wSl/MjFfe+7f7UXU9p1rWZJGJH8mda7Q4opYEqiTGatTeqpag8+Emcehw+TbdZGB4OpAieFkJthJ6ev4u7nrLF+ZH2GKyaaA9BpTReJTFGsTrww6j88YuVKEY9DhZpjHHMVAo+EzryKYLJQtCNwPOwEHLDAuy1ChZDyHSAaiWolMZrI4Ygp0N+Xrrj5gHwNB3jM3HHM4EjfL1qbvzvf2+7Y3x64CGm7wTRdw98FnSeFoJsWIBMBXVYDFFzB6xtgVIar5KYo8C/PIvN62svrtFzp5AbEO8o/fDFN8KCvrvl2vCsYL/2ppsf4npmfHpeUN39uV8bVLnHS0Kojc6N2V/ZQPz7xuCw9GthQfe8E3KeEUR3PN2xIe+tDz4UXw/cAfUt4/bn8lcHh18KO7ohOPziP8K87noqhZOwQz4Vgj9MBoiFN3+VCj5eEVb0v+s6W57m749+xcFDngRV1BD99s+/ssG/brrGLUjhM34L6uSSl7gfU29xP60UJ4v4VybRPMu3esJ0yPXvPTNOPcR0L+OfL02++uqkUG6HjHI7aSw0x+Tv0ZfgDiWbcjvZPMC2FCE/5AW69aKrP+YhnNrxLZ8/BVtRhoeV5pj8r+8fkluEEyHnfr7VM9PJD9TGNSp+JETrdfZ1pxJqTsKt4e6trFGJqx++iKljGJIraBbzvFqtgEDiKAMzoPUMuxTsyoBON8CX9WFmjmce+hT1qgkPa8wx+V/iH3LLZYQSWeuNDcDmJMhaVOArmZps5a6jeCOmvofHtoICCaOMzPDWM/xSUNDSbO7LCYwpZGw0x3N/f66nx2VQG9dm7d2GWrjEWh+g5iTAGujeihqVOIo3Yup5UBddHkhNlHVmShG6FPTKoAUyzFeUspEvVm7D27gqBt76NXqbRM1JsDWadsoAe+t0E3/E1PMURIkuYYWBRFHWmwGXggE73Shf6Zse1+ErEyu0M6F4/xIFNicB1viqZdyzwEk2+yOmPof1YCMRhYFEF0Jvhvei4fBON6yVTydLdHaY9FwigOYkSWSjV2Pqcq+adQkD9KLpwImi6Pd/EnWcbs2ajpICNypxEw/E1BP/AAJ3dmAgEXoz4FJodzpfCuLtRlT8pDUaaZv2AlD3VtSoxE38EdOk49AhKJAIrRnUiyZGJ/MOF/73SIMNfdPeGA+jddioxDbbY+KPmCYdd/4gGEj0yBXDDLsUBM0f1PHuFD8iPctjNO0FKcq7t8JGJdbZGXPHGzH1PCyOb45UUCAhWjO4F42uk3nHgF/kGr5Z1+Z/wH9WzgmlfuZmJzRrf2xExSFR98g/M9xSp/mUwOzMBFQ+8cgSFzssDA9HUgI70uoOKp90GuQzD1zRhP/8vw0HuvMcCw+rLqPgTHhYl+KYlbro+Ja4TA0Qm62QS+VkTYtZJ/wnuKzrzrMqJbCy68spS1+H6fpyrNvA3m4J0wNDX/bB3JRAFCCyjnVtnSrbsnb2pAR6M/cV7BP+c0UaToWHDV0KUskK9kkBYmsGQme6xCXFsThrZ36A6LPCiqyd3IT/dHxo1GdeWzca15qFxnVVd4oAyRo7NxrVkoVmNbdriV90uM+s9m6Mp8PWWLkwluYFFjdTw7BWmn4Ct0IOaVGvae3LG9YJvxEg2SGhk+V+p0fiYUsESApfamrOtHqR1ppnmpm5MogESGmClKxpOROy+nS3x8TsuCUt4E7CrO01rB15wzrgPhmJ5uTMa3Wvee3Km9cxKwRK83OGNbOnm68eAA==) **Figure: Remoteproc crash sequence** To enable and disable recovery using the recovery `debugfs` node from the user space, do the following: # remoteproc user space interface # Disable SSR: echo disabled > /sys/kernel/debug/remoteproc/remoteprocN/recovery Copy to clipboard ### Coredump Remote processor coredumps are supported for debugging the subsystem crash issues using the Qualcomm Linux kernel devcoredump feature. When a subsystem crash occurs, devcoredump exposes a snapshot of the memory of the recovering remoteproc in the `/sys/class/devcoredump/devcdN/data` node, and an associated crash `uevent` is sent to the user space. When the sysfs node is read, the sysfs provides the segments in an ELF container, and a write operation frees up the resources and destroys the devcoredump instance. A debugfs node is exposed to enable and disable coredump from the user space. # Disable coredumps: echo disabled > /sys/kernel/debug/remoteproc/remoteprocN/coredump # Enable coredumps: echo enabled > /sys/kernel/debug/remoteproc/remoteprocN/coredump # # N represents the index number of the remote processor. Copy to clipboard Note To collect the full RAM dump for crash analysis, Qualcomm SoCs implement kernel panic on remoteproc crash (fatal error on remote processor). The kernel panic mode is enabled only when the coredump is disabled. The devcoredump `/sys/class/devcoredump/devcdN/data` node is created on a subsystem crash only if coredump is enabled for the subsystem. To enable the coredump, run the `echo enabled > /sys/kernel/debug/remoteproc/remoteprocN/coredump` command. The `devcdN/data` node is temporary and is removed after a timeout. To debug the issues, use the `copy` command to copy the `devcdN/data` node to your local device for secure usage. The delete timeout is defined in `base/devcoredump.c`. The following is an example command to copy the coredumps: # Copy Coredump to a local device cp /sys/class/devcoredump/devcdN/data > /var/spool/crash/dump_file.elf # # N represents the index of coredump. It is incremented each time a new coredump is generated. Copy to clipboard To transfer the coredump file to a host device and debug the issues, use the Qualcomm Crash Analysis Portal (QCAP). Note For more information on QCAP, see [Parse RAM dumps using QCAP](https://docs.qualcomm.com/bundle/resource/topics/80-70014-12A/debugging_linux_kernel.html#parse_ram_dumps_using_qcap). ### Remoteproc configuration and firmware The remoteproc devices are configured in the respective device tree files using the DT bindings. The configuration includes compatible device names, memory regions, interrupt settings, and clock references, according to the DT bindings, in the `Documentation/devicetree/bindings/remoteproc/` directory. The following example shows the QCS6490 aDSP remoteproc device tree configuration in the `arch/arm64/boot/dts/qcom/qcs6490.dtsi` file: remoteproc_adsp: remoteproc@3000000 { compatible = "qcom,sc7280-adsp-pas"; reg = <0x0 0x03000000 0x0 0x100>; interrupts-extended = <&pdc 6 IRQ_TYPE_EDGE_RISING>, <&adsp_smp2p_in 0 IRQ_TYPE_EDGE_RISING>, <&adsp_smp2p_in 1 IRQ_TYPE_EDGE_RISING>, <&adsp_smp2p_in 2 IRQ_TYPE_EDGE_RISING>, <&adsp_smp2p_in 3 IRQ_TYPE_EDGE_RISING>, <&adsp_smp2p_in 7 IRQ_TYPE_EDGE_RISING>; interrupt-names = "wdog", "fatal", "ready", "handover", "stop-ack", "shutdown-ack"; clocks = <&rpmhcc RPMH_CXO_CLK>; clock-names = "xo"; }; Copy to clipboard You can get the device tree configuration file for QCS9075 in the `arch/arm64/boot/dts/qcom/sa8775p.dtsi` file. Note To get the device tree configuration file for all the Qualcomm SoCs, see the respective platform DTSI file. All firmware files are present in the `/lib/firmware` directory in `rootfs` and the related configurations are done in the corresponding board device tree `arch/arm64/boot/dts/qcom/qcs6490-rb3gen2.dts` file. For other Qualcomm hardware SoCs, see `arch/arm64/boot/dts/qcom/-.dts` files. The following example shows the remoteproc DT configuration: &remoteproc_adsp { firmware-name = "qcom/qcs6490/adsp.mdt"; status = "okay"; }; Copy to clipboard ### Sample logs The following log is displayed when a remoteproc boots successfully: ** \# remoteproc success** "remoteproc remoteproc1: remote processor 4080000.remoteproc is now up" Copy to clipboard The following log is displayed when a remoteproc firmware load fails: ** \# remoteproc failure** Error log: "remoteproc remoteproc0: Direct firmware load for qcom/qcs6490/modem.mdt failed with error -2" # Caused by: error loading firmware. # Solution: ensure that the modem firmware is copied to /lib/firmware/qcom/qcs6490 in rootfs. Copy to clipboard ## Memory The Qualcomm Linux kernel baseline supports all memory management features and allocators. The following information outlines customizing the memory map and performing heap management. For more information on Linux kernel memory management, see [Memory management](https://www.kernel.org/doc/html/next/core-api/index.html#memory-management). ### Memory map Memory map describes the areas that are reserved for subsystems, such as modem, camera, aDSP, and cDSP during kernel boot. The memory map sets `no-map` for the carved out regions in the DTSI, making them inaccessible to the kernel. Configurable carved-out regions are defined in the `arch/arm64/boot/dts/qcom/qcs6490.dtsi` file under the `reserved-memory` node. reserved-memory { cdsp_secure_heap_mem: cdsp-secure-heap@81800000 { reg = <0x0 0x81800000 0x0 0x1e00000>; no-map; }; camera_mem: camera@84300000 { reg = <0x0 0x84300000 0x0 0x500000>; no-map; }; wpss_mem: wpss@0x84800000 { reg = <0x0 0x84800000 0x0 0x1900000>; no-map; }; adsp_mem: adsp@86100000 { reg = <0x0 0x86100000 0x0 0x2800000>; no-map; }; }; Copy to clipboard Note For Qualcomm SoCs, see the SoC-specific Qualcomm DTSI files to get this information. The following early boot log shows carved out region creation for different subsystems: [ 0.000000] OF: reserved mem: 0x0000000081800000..0x00000000835fffff (30720 KiB) nomap non-reusable cdsp-secure-heap@81800000 [ 0.000000] OF: reserved mem: 0x0000000084300000..0x00000000847fffff (5120 KiB) nomap non-reusable camera@84300000 [ 0.000000] OF: reserved mem: 0x0000000084800000..0x00000000860fffff (25600 KiB) nomap non-reusable wpss@0x84800000 [ 0.000000] OF: reserved mem: 0x0000000086100000..0x00000000888fffff (40960 KiB) nomap non-reusable adsp@86100000 Copy to clipboard ### Contiguous memory allocator Qualcomm Linux distribution supports contiguous memory allocator (CMA) to allocate large physically contiguous memory. CMA reserves a large physically contiguous memory area at boot time and provides physically continuous memory to CMA allocation. When not in use, CMA memory is available to the kernel buddy allocator for movable allocations. To modify the size of the default CMA region, run `cma=size_in_MB` in the kernel command line arguments. For more information on how to use the kernel parameter, see the following example: cma=nn[MG]@[start[MG][-end[MG]]] [KNL,CMA] Sets the size of kernel global memory area for contiguous memory allocations and optionally the placement constraint by the physical address range of memory allocations. A value of 0 disables CMA altogether. For more information, see kernel/dma/contiguous.c Copy to clipboard The custom CMA regions are defined under the `reserved-memory` node with a `shared-dma-pool` compatible tag indicating the CMA region: adsp_heap_mem: adsp-heap { compatible = "shared-dma-pool"; alloc-ranges = <0x0 0x00000000 0x0 0xffffffff>; reusable; alignment = <0x0 0x400000>; size = <0x0 0xc00000>; }; Copy to clipboard The following is the sample log for an aDSP CMA memory region reserved on boot: [ 0.000000] OF: reserved mem: initialized node adsp-heap, compatible id shared-dma-pool [ 0.000000] OF: reserved mem: 0x00000000ff000000..0x00000000ffbfffff (12288 KiB) map reusable adsp-heap Copy to clipboard ### DMA-BUF heaps DMA-BUF heaps are supported on the Qualcomm Linux distribution to allocate custom CMA heaps. With DMA-BUF heaps, a device file is present for each heap in the `/dev/dma_heap` file system. Apart from the system heap and the common CMA reserved heap, you can create your own CMA-type DMA-BUF heaps. ### Supported heaps The following table lists the heaps that are supported by default on the Qualcomm Linux distribution: Default supported heaps | Heap name | Dev node | Description | Usage | | --- | --- | --- | --- | | System | /dev/dma_heap/system
Copy to clipboard | The kernel creates the default DMA-BUF heap. | All generic use cases must follow the system heap that uses the common Linux memory management buddy allocator underneath. | | Reserved | /dev/dma_heap/reserved
Copy to clipboard | The default CMA-type heap created in the system uses the default *reserved* CMA region. | If you need contiguous memory due to any constraints, use the CMA heap. | | Custom CMA heaps | /dev/dma_heap/my_cma_heap
Copy to clipboard | User-defined CMA-type heaps. | If you want to create your own CMA-type heap for specific custom CMA heaps. | ### To use DMA-BUF heaps The following is a sample program that demonstrates how to use the DMA-BUF system heap created by the Qualcomm Linux kernel in `/dev/dma_heap/system:` include include include include include include include #include define DMA_HEAP_NAME "system" define SZ_4 0x00000004 // to allocate a 4K buffer int main() { int fd, dma_buf_fd; struct dma_heap_allocation_data dma_alloc_data = { .len = SZ_4, .fd_flags = O_RDWR | O_CLOEXEC, }; struct dma_buf_sync sync_start = { .flags = DMA_BUF_SYNC_START, }; struct dma_buf_sync sync_end = { .flags = DMA_BUF_SYNC_END, }; fd = open("/dev/dma_heap/system", O_RDWR); if (fd < 0) { perror("open"); return errno; } dma_buf_fd = ioctl(fd, DMA_HEAP_IOCTL_ALLOC, &dma_alloc_data); if (dma_buf_fd < 0) { perror("ioctl"); return errno; } printf("Allocated DMA buffer with fd %d\n", dma_buf_fd); if (ioctl(dma_buf_fd, DMA_BUF_IOCTL_SYNC, &sync_start)) { perror("ioctl DMA_BUF_IOCTL_SYNC start"); return errno; } // Do something with the buffer here if (ioctl(dma_buf_fd, DMA_BUF_IOCTL_SYNC, &sync_end)) { perror("ioctl DMA_BUF_IOCTL_SYNC end"); return errno; } if (close(dma_buf_fd)) { perror("close"); return errno; } if (close(fd)) { perror("close"); return errno; } return 0; } Copy to clipboard ## Scheduler The scheduler decides the order in which the processes must run. The scheduler runs the processes from the runqueue of each CPU. The kernel baseline supports the standard Linux scheduler solution. The kernel uses [Energy Aware Scheduling (EAS)](https://www.kernel.org/doc/html/next/scheduler/sched-energy.html) to choose the right CPU for task placement based on the CPU energy consumption. EAS overrides the completely fair scheduler (CFS) task wake-up balancing code. It uses the energy model (EM) of the CPUs and the per entity load tracking (PELT) signals to choose an energy-efficient target CPU during wake-up balance in a system with asymmetric CPU topology. For basic scheduler documentation, see [Scheduler](https://www.kernel.org/doc/html/next/scheduler/index.html). ### CPU topology and EAS The Qualcomm SoCs have a heterogeneous CPU topology that is differentiated in terms of CPU capacity metrics used in EAS. EAS uses the concept of *capacity* to differentiate CPUs with different computing capabilities. The capacity of a CPU represents the amount of work it finishes when running at its highest frequency compared to the most capable CPU of the system. Capacity values are normalized in a 1024 range (the most powerful CPU/cluster is configured at 1024). EAS builds the capacity of a cluster based on the Dhrystone million instructions per second (DMIPS) value specified in the CPU node and the maximum frequency supported by the cluster. The following are the `sysfs` nodes for CPU topology and capacity: > > > Table: CPU topology and capacity > > > | Commands | Purpose | > | --- | --- | > | cat /sys/devices/system/cpu/cpu*/cpu_capacity
Copy to clipboard | To get the capacity associated with each CPU in the system. | > | cat /sys/devices/system/cpu/cpufreq/policy*/related_cpus
Copy to clipboard | To get a list of the CPUs associated with each cluster in the system. | For more information on how a scheduler uses CPU capacity, see [Capacity Aware Scheduling](https://www.kernel.org/doc/html/next/scheduler/sched-capacity.html). ### Per entity load tracking Per entity load tracking (PELT) is the mechanism used to track load accounting for `sched_entities/groups/runqueues`. The PELT clock multiplier can be tuned to meet power and performance requirements. The command line parameter `sched_pelt_multiplier` allows you to set a clock multiplier. The clock multiplier tunes the PELT ramp up/down speed to tune the system for power and performance. With lower half-life time, the CPU utilization of tasks is accumulated faster. The tasks are placed on performance CPUs, thereby leading to better performance with high power consumption. The CPU utilization also gets accumulated to select the higher CPU frequency within less time. Note The half-life time also affects how the task utilization ramps down. The UTIL EST feature provides faster ramp down. For more information on PELT, see [Per-entity load tracking](https://lwn.net/Articles/531853/). ### SchedUtil governor The SchedUtil CPU frequency governor is the default governor in the Qualcomm Linux kernel. SchedUtil predicts optimal operating points (OPPs) based on the CPU utilization, maintaining coherence between frequency requests and energy predictions. The SchedUtil CPU frequency governor is tied to EAS, and it tries to predict at which OPP all the CPUs run next to estimate their energy consumption. For more information, see [Schedutil](https://www.kernel.org/doc/html/next/scheduler/schedutil.html). ### Utilization clamping (UCLAMP) The UCLAMP operation allows performance management of tasks/task groups from the user space. UCLAMP allows the placement of tasks or task groups and allows performance hints or constraints. This mechanism is used to influence the scheduler placement decisions or to influence the frequency guidance of the cluster. For more information, see [UCLAMP](https://www.kernel.org/doc/html/next/scheduler/sched-util-clamp.html). ## Dynamic voltage and frequency scaling (DVFS) The Qualcomm Linux kernel baseline supports several DVFS implementations to manage frequency scaling in a dynamic way corresponding to the system requirement. ### CPU DVFS You can select a governor to tune the system for power or performance while changing the `CPUfreq` governor (`powersave/performace/schedutil`). By default, the CPU frequency governor is set to `performance` on Qualcomm Linux kernel. Table: CPU DVFS variables | Variable | Description | Path | | --- | --- | --- | | scaling_governor
Copy to clipboard | Set the governor to performance. | echo performance > /sys/devices/system/cpu/cpufreq/policy*/scaling_governor
Copy to clipboard | | scaling_max_freq
Copy to clipboard | Set the maximum frequency of the CPU cluster. | /sys/devices/system/cpu/cpufreq/policy*/scaling_max_freq
Copy to clipboard | | scaling_min_freq
Copy to clipboard | Set the minimum frequency of the CPU cluster. | /sys/devices/system/cpu/cpufreq/policy*/scaling_min_freq
Copy to clipboard | For more information on `CPUfreq`, see [https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt](https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt). ### Cache and memory DVFS The static map DVFS governors align the CPU operating frequencies with the following: - Level 3 cache (L3) - Last level cache controller (LLCC) - DDR RAM If the CPU frequency is at the maximum level, either by setting the CPU frequency governor to performance, or due to load in the system, then the L3, LLCC, and DDR also run at their maximum frequencies. The driver for this governor is present in the `drivers/cpufreq/qcom-cpufreq-hw.c` file, and static mapping is present in the `arch/arm64/boot/dts/qcom/sc7280.dtsi` file. Note For Qualcomm SoCs, see the platform-specific DTSI files to get this information. The other bandwidth monitor (BWMON) governor is used to vote for LLCC and DDR frequencies, based on the measured traffic between CPU to LLCC and CPU to DDR. The driver for this governor is present in the `drivers/soc/qcom/icc-bwmon.c` file. ## Virtualization Virtualization uses software to create an abstraction layer over computer hardware. The abstraction layer allows the hardware elements of a single computer, such as processor, memory, and storage to be divided into multiple virtual computers, called as virtual machines (VMs). Each VM runs its own OS and behaves like an independent computer, though it is running on the same underlying computer hardware. ![../_images/virtualization-3.png](data:image/png;base64,UklGRogOAABXRUJQVlA4THsOAAAvYIJ0AN8Et7ZtVdXC3aEAdwspgbYpgJDIc/kf51xvw41t22pycHeo4wOAiqkwVBYToKJzp/YkEHkxx7VtK82Je0aZuWsFaTu9uJeA8//nwZv/AIZMqSCjJBXSEgkN45X5fCJkKlYs+/lGSGHjTED5JGcTocUJUmcTEfKWS4URIYERMqU8JUGKolCBoX31Fd/7F3Jx4fOzbGLcbY/4mUgud7d9OhQ+E0aES4WLCzZ27rbHJcpEjgqMu+GhDDRQxxIYD5hItDEGMkC6DziRt0i/qsQRsJdu1w0BFjDRBY+6wx0lu+7RmWkCFMAKd1RHbcV6AhYaXhcGDgDe2oAzOK3AygFgog0ujwBXnAFXVNt98LYHWHi8IywAX0eQr7fBytuLsEe+3AVvb0nsQLrcBi8iKXhKl+u2APZYwpBK8AcXBrEBUAVrUMcUcIEIxlVLYDyRmFStBgINebsKE13jiPNeB2CAOYrvoeerbb8DrMDKwwus6wPbNYxgsToTEvDWnQEe//97mzbbv0+O8zx97n12771bhDij0+5uoc1OVJSYCoK7SVH3MCTI6i6VU58DcxK5/W+Pz5Aw9iFzZCii/5IgSZLbpsYXaACqxQA7yyUp4X3gF7YrDhz4i+0KS8aff2Ov4k+UPyp7Fb+r/Kfyn30OgZs0BYV4px0KvFIY32uHBdiV49Bsu3FLqQaTC9z+SFFiGZr/BiD+MDTdHw2IGCm850m8+EVzzEi1l+/RYxDMwGhRXhnqOYCx3jQcPYcdR9+LW3Iak5FheKdNJnSYFx0YMUorQ74HW3oFDyJzkZelnpOpscX3gEem/O9g/fkRo7QyFLik+XpPplslhYhVnj45ur6XpRA7AJDRo023m90BiFdCmUb71NMQn1P+0wAfSmrI/AzA5wvIKU+rUVv9+I8zaEHMniaSRQi+Xgm7I0Z5ZUgpQxK47ZDtiXHegfWZXFIgEeXxL3TZ8oiRpMZHJUnT90gDIGLkm4GQky52Pw2uMWqUVYYED+MtKNWIW+z4MCBiBUcicJMFpXo6szLQXlAXHdxuFtqnZTVsxA/YwaDUi5BxMkx8Rpzf035PQj9vzRGlpDIkj0hNhjSU8T2WHmJQAC3yhxue24E84hYTJjKCGlYLabgkdTWoyPqfnpNRryxvzRGl3DLUc+KWMhz0gPOcsXQUgBWIXHiiKEyNSEaIJEXJbQazzB3RjCSllqEU1hdM+ApCCn28tndH4MKnK28cd3fFaaY2bJLmOFBaGSIWumUmJHAzHDcXQaoQ9EwLdscsU6MFZ3eICS0yYh5JyipD5HdkytdAG8nL0Fe5kBdCyDgakCzEZ/K7Q1BjU48XnWwX+F58TjvMOlnOiHlUKa8MNaCvTEnPgaSZS4qdfxd0rM9gweH1YTcFR1Jj9SRuFYeYUTUlDJ17Tn9UKasMyUXbbAh9fC4BubJfdOCjU16bXOhwfkt3ErsglNQU0VxfUMVh05bw0QIveRF58XhUKa0MdUHAgr0PrfKfyn+uXOC2OyaDGArXTQgJum4PeBQmgxgKd8BkEHfYSR5QpR93D4/rVNnH7yv/qfynOp9bSuRTOKNPiXywy0Zz9x0lEHeNCY/eUQbxyFhyXSmMWcaEB8phGFz5T+U/lf/sNWncIvScrFTZ6wRd4wL/SQfgswVk/mmAeCVUvpelkCxoZK1GcuZJpjC7QwRRdQbgc1naZBgnQSVKAzDQpOcQhUxz3oF2qJG0kP8zhRRIRFw1aUrSJsM4CSpPAhcTMwsRpua9UPlPQt/3tAaJPK34ddTKAhc7i542IoL+D9AXpQ2GeRJUosQf3fCqOMDxvcz3kiZF0tIQW+r0hxue24F2GLjYAeRJGwzzJKg8F1IAOhbpOUCjfcpj5UaJWqqRNJk0NdT0nIgiSofmWjBQgsoQeriPP00PvkByRkDUkvivC5+uvHHcLUjSNBcGSlCZ0WUjlpQPQF65OBO3pIPvC4haUsV5megGbl7Ficy3IwMlqMwIXDLAfdFJmlh1XlVq3otbvhefw/Gxk0kIWgisL5AxcxeyEJ/ukzHzllIXnUxJ0gbDPAkqNVQKNLaEp/WQCVUjbgkdQo5WI96hJSVweXFhqnFLlDYY5klQuaEufqP57HWl8V9wAM5v8alH+GhBcUQtOm/5tNYKcbISVXQFogrrC7K0wTBPggzKmAShafi3CRknQVf/YcHeql35T+U/1X/cdl0JxM1jwiPXlUHcZ8Hu+slC8+vrSiN+P/pcVx4xmpRjjAqlGJX/XMuU6x8ojbhl9HmgPOLKr/ehWbAPm1T+U/lP5T9XPuJ77bCUGNkMTQAELujoTwIML0MTFylESqkuZEOiG7dUbgzP14AYJEMlQ8+J6PGEfvkzzAxNVjSSplBE0vZpFyCi2QPSQ6TJvx00aQAK+R6gDyk4giSVYgpELWnq1tAcUY8z1FdqTqP9owPQR+PIIBgwQ2bG99qhkLcU9PHtQkR6B9pToApKURd2XVNApySFPs14GrdYOzApGe1dGu2QeZD+QGxOA6UbAH3sh8yBCTNkZAI3UxI5B1mlJB+RqJgDGgRuhI8TASYXsayJUor5Cs2hvVAXiHBmDkyYoRIAywTmoef06SFm1zm71HPQksRFgMmR/KIvrzeRgNAchGn4nkEwYIbMjO8JtEMRwJDwyZCkIELB4f4ASRPLCraEe1BYc8yFCTNk7FWEjoGRNPHCJUFg+fG9ovgeTYEQPSdparXTbiR5kP5AaI6pMFOGSgBhyJoKF3+jHeryIUOGubug92Urvy2kDc/HLcmDoJtjNkyTofJANTBfdGhMn5OhBxk8d9shwsc9DchI4ooUHNBBsk5cqVbgQjuUPHDySmqOqTBThsoAOtMDSZNPIUGfyeDhpiMVOhaJGhk54pDlQK5yJki9gI1k+pIH9RWaYyrMlCEzMrQgNBWLPbDtOf0CQyXjvshuwgyVNV0gl3WOM50tMi5jnaEJB9UFUmRy2xC3zMs4Z8iCvdGx8p/KfyrvsWA3ZWShuf6B0ohbRp8HyiNGk3KMUaEUo/Kfyn8q/6n854pUnnig9OPx4fFI6Sfoob1iEoihMQnE3nDvA5NBDIVbJoQE3bJ7LFBU/lP5T+U/lf+MH7fdYWniV0P9tlGWJn5vF3nkOvsSdw+bB5R9iTsq/6kOI3CTpqAQ77RDgVcK43vtEB+XyZFlUrvS3mf/0YCIkcJ7nsSLXzSHiShrIeg57ETzvbgln/BJc2hnZjFZW0De+dxzMmWnUClbbECfnHS+l6UQOwCQ0ZOQbje7AxCvhNJCF0jELcHJ91CWec4/7QB8tIAWP85AfI6S34ouaXTgZihKG0YbnizsIwncdsj2xDjvwPpMLimQiAohOMn0HOKSNH2PGRNyW0G3VV3oc9FM0Ya3w/3kjhq4SPowBkSsykoELnZlPbcdCqsIT00kJ7JAacCHWvFJyDQZnsmZSK4pWa8B7xHRPm/4fpKeQ7ordiITgwJokT/c8NwO5JLCh0pxJxnSHZPWnCHNYB4qvxW4Pq5OR55kNdLwfSV0XbIqpx0WgFXOPFh/yJxy6Dm080vj19jFISOZ4v8Jy8wFHfebryCk0MeNdkfgwqcrbxx3c+gCMeROw0Q0VWnS1LYSSXM/SuBmWDoLIZTVLvRRUIb/Vtyp8EIhmCmh5/zPzZRGGO/tPyEn8cvQVznQrrRBnnEAkoVqfgZEeg4boklO2P8WWKUQkil9rton04DncNrEyfan9BxImrmkADgq07E+g8WUF07OKY8P1k5LTmn+juJWEeRWYAPJniTRfSl0QiKXgHRsLzrw0SmvTfo5OL+le8YihIITlUVxNl/74auqADmtEF65IKJkRngyYMKMLvSLvERllZj32qHVAkeBkbJa+B58aMHeql35T/UW1vGHcFmZsIz8/TpLE7+0YD+MvvKfyn8q/6m4w4J9Vx+7yGN32pd4cNg8WLMvcWflP5X/5LK5sUw4MjUoBne6phYvXQLYeGexVjt5cLAJnbO1o98CfHB46qeDa1pmDlbRaG1REJT8niHC9oNNILGKXJiCtcUjU+hxYWpA9Oqg3U5Mr0qCeX4wsBMAj4E+Z3UfWDsyvbaooZvC+4u1Q1MwmINtbfFwZ6k2t7EsClK/Wh39Tj4F27YDHX+76dlLQOgsCV9kDgYnpgfa4uevlmv1zpIoKPthnbYeC7TCUtYWkVUEH6mvLW52/jO9jQ6iIPcDYMZWA1yAT95589h0MeY6S3W96eDI1HZNECxAZ8lyMEfGbic48oJWeFt/tc217/W5ywVFVvmOrAeDxdrRf4JArc5XwW30SnMblwa1miDI0FIb7+KcydSgZj0WeIGl8Jr7CZk60TraDLcXBXP8YGPZdoCdIVx4q76xzNETt2SCd8DETh5ETVFQ9oOPz1qwtwlV/lP5z9WDXFH3cyrvfcDSxK8t2M8Kr/yn8p+rJrBbYcn4re2KA9fejn/cPo6xz+Z2GMe4AoVbbxyvuH8/zl9r4xX3XKXFXt3q1DW1eOkSwMY75MZZBpvQOVs7+i25SY2fyL1ozJH769IKgqDk9wwRth9sAolV5MIUrC3S22e5MDUgenXQbiemVyXBPD8Y2AmAB97Hje4Da0fonSjRTfmNAs3BNt4rUGepNrexLAqiH7+hoJNPwbbtQMffbnr2EhA6S/KtTp2YHmiLn79artU7S6Kg7Id12nossAor3kvXKoKP1NcWNzv/md5GB1GQ+wEwY6sBLsAn77x5bLoYc52lut50cGRquyYIFqCzZDmYozfKxZEXtMLb+qttrn2vz10uKLLKd2Q9GCzibXIJ1Op8FdxGrzS3cWlQqwmCnJMHN97FOZOpQc16LPACS+E19xMydaJ1tBluLwrm+MHGsu0AO0O48FZ9Y5mjJ27JBO+AiZ08iJqioOwHH5+1YG8TulY/FuzzmBbsE+VWDBv2XX2uCRYHAAA=) **Figure: Virtualization** Note Virtualization is supported on the custom BSP variant only. ### Qualcomm virtualization solution Qualcomm uses Gunyah, a type-1 hypervisor designed for strong security, high performance, OS independence, and higher CPU privilege level. Gunyah does not depend on any lower-privileged OS kernel or code for its core functionality to offer increased security. Gunyah is designed for isolated virtual machine use cases, and for launching isolated virtual machines from a relatively less trusted host virtual machine. The following figure shows the Qualcomm virtualization solution architecture: ![../_images/virtualization-solution.png](data:image/png;base64,UklGRswMAABXRUJQVlA4TMAMAAAvlgJ4AJ8FObJtVdW65777Dhdn5C4BaP5ZOHzXEe6WBhzZtk1b8/ghtN2CG77wNeU1WKGtj8jXF3JtW8ua/ev7cCdzpARaSdMUEPcRs3iCz3+Az+xMbTAyoy4xImNFwZqCiJKzJSNQG5xKRJ8FIrEWIzKhrYnMWF0QmSErqAvqgrok4oI1BDJRG6xFZMZkNXVBohZXPEDtqe4wek9VmzEHoWyLya2b6bLuKKCJ/7Z7csQLHnzD1f8FM+6zAm++AF6V4Fp+53If8CgCL0T2PG59301CG/Zxh0ubBkAdGhxF7SH/XP9zB5uBVwXwX+8YQA9Y4MyNC8TJOdgSv4cnvPgBkXuPaoU54u/BTH0MA2rAAHrcxJe7XzsH/87B1jn4d2sf1AAecYYjb8y8Dh7BvRfgCNxyvz9ocC21NIAej/mFt9XGf/qDOnCHFxsZlukWD7jxBr4cGANr2ZTMgeUnb0Ab78Oa4y9DoAglKAMrrGEObMEStPTgyA+Y8fRk6cswhkfytwlevQO8eAN45Ist2d8fTICZTHuDwZU2cIQps+PaeLZyZKvjXWqzADMUxW3bONL+YyfX6zMiJoA/qZZsb+MzWgbLDVZy2Q7EWAOBS3ZqrtN0N2VJ2vY4UqPxBGhyzlBDzjtfwKtxbclpNW6mj+BbeGdvfVcyXb9ElRhaEf2XxEiSIknmy9zVu3fVg/X2rm3b27a1tp2LkNXoVSW9Su42nYxxg7QYUAR+BUyv5Ig5LDdihHQMiGMAoESCYsT0IrhwDe/7ui/QBSwYLaL/EiXJjttmnnM8FY+BjEcCIL+g9utEIas91JzSx4Z2H1BXH3Ob5r9KUFsDe7K7H51MVMM0Kbs9ok6ayqS3Q+TNf+Y/85/5z/xn/jP/mf/Mf18NFR4xDbJFkDVy/TGJibwGaTi4JpVIBnhU9eHYaBDb5vPQFolvCzGsTozrBNYjzhURjGhgkmfDRD6wgiib5FkWyxa4zlQhOK4njpv/zH+Vc+rQYygdTAbncx/Bc1b+SEJd8jzM6Rzq2Nu2z9+AmTeSBBXbtgv+cDnFaQUV/hYd3NlOkRCM0xpY2pGWUgo+Jmkjx6CSIiGXWpceN+lVl1KS3j96VbajOb9jmCLBX0ptLs/UusOvYTJod+MUiaQANSmUTh1mrTuwNPT+lSKRFK9gTQpl47R6VTWVvre/kuJDsCaFsin4E7K2w/H+JUxSJNLiC4FNCkWTBBW6OXtfSZVIi5PxwSaFkql1QWPCEBxYNi8hzJmevSrjYM7nh55gk0LJ1KExsFKuhayEaJw/5u8rImCTQrFox1aRwJyC5eSFOqxJoWQK/pARWJLAmhRK5ggmt+a7HAE1KZRMrdurMoUxWY7AmBSKf4G6NBFkUqhjwkwKVUyYSaGOPYHI/Gf++0pOotVqKRHElIBQKyHUABG8bkyiWc+zFQgLnmcLjyTqw8E3BHyLYxlnp4HlDcbq9xHs1PHtqNTZrzQjdtqNRzJVLmky/5n/zH/mP/Of+c/8Z/4z/5kIcs+1IqhTz7UQJNi9AHoesE4JzhrENhfQArFtoFc+EXYvwP87kA3n7+aIryygOvFtJ5w2590b3Cxh5suw7FjWCIpo1LFsLggBdi+ASWo4uCZLUjA9pApsgelhVBAC7F4IGeBRlfnP/FcNpw4at56zbDv56S2L4Hx6bTLr+OQagJ8upEhkDoyhkkYdegylg8lT65rZBBhKAFO3mXE4v4PGHUuiSxxkDZChElPtBrKG1sxP/KBiJ8F3Q0Jyy59xJNi7YMkv64Yq4oIfqKvduRswMwbKUEnDnMJB65XIIJxA0vuzVyV3ihdnHAV/oA1yWjDtI5iazTBjoAyViJzr2Ptbxjnrm8qhd3bGkQQVppeljL4yBspQiaj2z+eseraT3oddVQbJ4Nxsw2kNLFbBaeHOBSuLAAyViBiNDKLHgS2pJgNL5pPCbKPWHWoO5KuynUvI5m4gQyXVhwSVmcGt92xCXX0TQWXGYtvOx799BwysjIEyVPHCtO3zreB5myK3kLlVmH3JVAp/IGuboAxVvLxi0mnphCJ/WR/IDwqzdJNal5lb96rZ2gRmqKKGXesGFXpbOwpe+1cliwQ9lScJKhkl9FBFEruOgUUo+I+o/WbhMr6tr4nXETxnqRFQ5hKYoYocdoIJnZuqX5Fkpj70PAKNvWrGHnqiDJVQH4JelXEwM4z+EZmbJjAVs/UVkXeu+fQll4yBMlSSyi3f8/y//5j/zH+f6DRz6SZqudAdtVwWlFqu7c38V1VqnGZld0R0mqb84pns8pV2N/JEbOa/qhIdl7jt5OHtOcDaxfsdUPI2F9wKQVbvvf9GZpIXwm0fod7ITBnWf6RsLQgDHq6Fkhyfp2yDQt2AofnP/Ffp6NHLHnB73bivHOv0jek17rv+4sJvet5QOpgM568BwYE1/zi2AxpheTmi4BOXvercw8XtVUOIUzuIig+WhTN5+V9mFqeF9+QGLUzmHQ1vX6uCJlaWIWrdoXak+UV6EZz59RnardV+artX2gjXhPgSI2W1G0u+uAmE68RqsyOtKJwGb+lGGzvAjyXJ6ctEVwnGTUTtsEzc+pukhTse0SN+HjAuaXqlBQMmZjrB/rlNASRG2r5fqrJpai6S/TYxILAanKXkW22o3PCIbqy47aH/BUbU7ZjC1UfUul9UpHrGIkN9ziW0905PbIgmpJYbFcl2UlJdORu7fUNa9UviImIKp8FaugifVRO5WBp8byiDFSmolwxxU8jDGkKQCnFxaBBTpbdAbBJUpqDgD6z5yJldyOhv8qhuTfrql1hhyuapN365CUVYplYEVoOzdMMyUYpe9WIqF9PW6LUVC0o4fmNLCNoMaroh9RYaCr50mYPJtDAKquOSXL+7S3QrCi/JWbq0SzfDlzwgxVd5kc+JRhOgI96GNxLU9BnaoIUlOcLe9rxbxifXUma+NDWiGRVdes3Gz+tvP7p7b+iW00FHC/SCzTbu0cu0VhYlnBYGlj3/aHh9g0MbMaTS8F7gDiDO3CO6ZUqiDVA506aSYcVPvdwOy2yyyOCQpdI5iHARrm8JcezlNvqGS2b3SEVNyFcIsSFOt3FvcJYu9kvaxClcU7MpL2ZoeFfCMqXhxbLZp+WomN1kkSGBOS+fxe9Ai/2S+JKcoJ2KaCIqkpGvPgi4FzhLN7xJJfWhcFjW0UpOsAdCLDi/RYHn9aqMg1nwGZn59ooIs3gejp8WMi56GG920ml4MTVsA7dX3bB8L3CWbrRxGRgb2jI+xiXWtImR0Dj2jadaI/Tl/sMtschQx0PDBxA9yf22fOpSuy7nebrTN+63pX7RRrKj+26pYo51cHjfLZ9TyytrmP/Mf5XCRsZStm8Q6pNwZWVoU/I2HNwsYdZ5t9546mpsBeczzLVKqeWS88x/5j/zX8WRfMulboXQtih5GxeaS1+Z3IpuypaZ/8x/BppPrgH4aVtSB4ngzrbkKKgQCv4wl8NpgcaEAQgq+R8J9j6ybedrP6hITFIhv2OY+1HHwNIEhjq20xpYeR+J1ult57OPGOykV83vYK7HzCU7T/yr1h3aU/D+JUzsnBEtZJ4DJlMQHEiXHHATzoDZxE5hksdR8AcWI5BCHRN6Fph53TL+Bdt2Pvdh2inUurhr2c5Xfq+a9z30JLApPe1O6S4i4nz9CICfPrIJ07w7cq2WF6gz/5n/zH99s5K37HNp/UX5ymTiNufVcm1v5j/zXx8er48tPFqdVshOasZwh5uM+6kaZohxS/1ZBs95dtwH5uWrAH58Nm/k2A5ojI18ERf7zxpy8zbiXJGG1zfoaTpYyRNxw7LQCyMWbrSxA/xYUpy+DITrhqyT/itthGscx3Y8Te70ZQ8Yl4SSM+h+xzpxE1FJnG4DP7y8AZri+a8xo+BG30KGKpGGBxWxunyTGDG4oHK6W1RM4baHvtHwqNFLG6ArzuzGXC+Pii7CZ9XkS8q4+N5QCiuS2BBNxBLNVXX9LzCSbocG0UxBFQ7Z79jf6fuLG6A6wjK5XkdFNZQgCrGyoDXAEI7f2NLqQtPkiYp8e17aAN15pShkNMNywwON/ibT08WX9ExNkq8bDW8kNE0OaiCVNF7agAvF+ZCpiDamQDx6mfTvV+4z/y4V5wtxCdefj3Gwcerldvg6l+gjDEpa8l9vTsYXdBmfSJT0uVO4piZUXjwFDS/eEuJ0Jyxrm1DjkaooDaWwKhvz5+UN0JwzPd3wZup6Ej8uSDsQYs5N0HHFfltD1/57++IGaM5FRNxo47L+VP70ZZAl+mk49o0H3F7VV+gPt4SMix7Gmx0Gsoy/f/XiBqjNBeooxYf4oWf1HyFE7gfb+rSszTBuu2aJVwM=) **Figure: Qualcomm virtualization solution architecture** Gunyah provides the following features: Table: Features of Gunyah | Features | Description | | --- | --- | | Threads and scheduling | Scheduler schedules virtual CPUs (VCPUs) on physical CPUs and allows time-sharing of the CPUs. | | Memory management | Gunyah tracks memory ownership and memory usage under its control. Gunyah provides low-level dynamic memory management APIs on top of which the higher-level donation, lending, and sharing are built. | | Interrupt virtualization | Gunyah manages interrupts that are routed directly to the assigned VM. | | Inter-VM communication | Message queues and doorbells provide mechanisms for communication between the VMs. | | Device virtualization | Para-virtualization of devices is supported using inter-VM communication and virtio primitives. The hardware virtualization and emulation support low-level architecture features and devices, such as CPU timers and interrupt controllers. | | Resource manager | Gunyah supports a root VM that initially owns all the VM memory and input/output resources. The Gunyah resource manager is the default-bundled root VM and provides high-level services including dynamic VM management, secure memory donation, lending, and sharing. | The Gunyah hypervisor support is added to the [crosvm](https://chromium.googlesource.com/chromiumos/platform/crosvm) virtual memory monitor (VMM). The crosvm interacts with the Gunyah VM manager in the kernel through the `/dev/gunyah` interface and creates a VM. The following figure shows the blocks in the Qualcomm virtualization solution: ![../_images/virtualization_solution_new.png](data:image/png;base64,UklGRkAjAABXRUJQVlA4TDQjAAAvlMKQADWL4rZtHGv/sdOvfiNiAnRu5iPS2pUZYvOPuCslZ5TQPFoo1L+TdVgj3bMqcY42oTpFg9YFScAKLhKrLAywBkKrvDp2Ed+K+/+W7DgDDQ0NBxoOHGhoaDjQ0NBwoKHhQENDwwuvBgUaBl4YGGgYFqm76vSp1+lz/jO+dwe9hJaGp/KmLfOororOo/PgtQGrFlBjDW21vIFaQkteQG+gNJ0VdOjL0xoe1QLyODsoNmjQsI5mA1fZQC0gj/7ngWsJPQsYVd4v66p3UCzvpBT6fj9ag8KOZJhXM2ojyZEhHJyFcPyRbLhQTzeNJElSmGXm3jsQF/LC6b8kto0kSaJ7ZqfnqPXevZ2ZlfXG42z/7EZ2tBwtQUvQUrgUhbUELYNL0W4KOP/v9/2PVNVDHU+pgMo9T6zUEwN5rMq67xT9so7uwA14BVOmH+GsgCrPEaOu1BE9duRcfxDKlBBQVAA9nLCjGzZRkIc/1EllN6MnRvIo49YCOvUOGhAIHA8b8AqYXo+KWI6YegMnJyqq7LYI3CGkxwX06LS2trD5ZCQS+bt8EskGRUZ2hI7UESIjE/dLJPKL+0dAIp3W2hNWv4xEVtYRGdkNHjKyI3SkjoBE8jaIRAaHjERW9p8WJElCW6l41gXygoq4AQ/LMuj7zUBn3sBDnOMMJ4C7nMY9z+uhcZHj3MtZ3Au8DrhL6T7IfSj3Ci1cRo13gMCHCK66BNEsDxAGkz3AkvRyijeBYJiJq/9oHiIgZjHTCNUbQSBEPt60QFjMbNF0lAyBMSpATHOERt0e4nICjrboII5LcFTFiCfZBh7nfijEPN3z357/BFedAkIkI1TuwT0cIweEHETzsRc63UzFrgSHMxqUnfkkc7JVvAiAYp3uUk2dazmqAOMEMEWeOVO4mCBoJI2lI41m4PHUUiwdqJgiwsctc5zZAgEkq5HT1CpECEkxdZjFDEFErUJHqWIEkiwnjrKYQYlsGDpIOMwQSjJaOUiaI5goxw5STOFENjcdhH0EEY8QOgWwzJZXgXOoAFDkqXNESGNjXyvFzFZwb0QuLmOufK3y10ut2hKep8WDCsdShVll8NOEVuUYC7MVyNOicrxLuDbunHAZ/6UfVvp9+cX+weV3PrbykkrRemxWGTrLtNjY+8LMLF5+0Npd6VjYwSzRQI+i2AeD8qz6aXbpLAk3rXr9iv2DCl+nUr3OXTOz7SyL1q5tF61LrHYEAkWx7w69retWh0NyUVSt2fbxXt0JxYtE6uBpUTVfuOU+YwezDIsk0uVeES8wORASx3vljekGcoFtwc8lEFKwtZtpcb7YlhzMmMZheVYcwvwl8cwtBe8X+3LZLI9iA+uxmdnBrLVbd1Zesq4ZhmcGbDZ23P5Gd6yOG9GTnSRLo7VrbxXF7QUcgz1Nzhf1DxXFg92E80UFozxLQ/a1ovLMgRroIVdkddhjDmZpyLw67Ve6PCXR/eCxWfm5D7mD9yX6wuxr34LU1raLTAt7DW9+jf/szWyHpas9Q06VE6LpsSL1qMzpUZJ61HQ9lqQeSNejINRYUPZYEHoUlD1yQpbm9KhJPcrWPbz/Gyz4ZeE9QaGiYT5KNR1cQirtNleEBOb0SEk9rkDXwyP1QLoeJaHGlLLHlNCjpOzhEuKZ0yMj9Uhb9kjxv8GaXxZXFQGfqbuRKlLk1VSFioqawyzXykO+TTmc9/jLasrDW5VF3RAivyyd308GkQcK1IKjBBx9kxz1ylvXAPmkfTgs0EY+9MPPaXz9X283mQ3Wfr5FNE3XdTvw0P3cM+4/i6Vs7aJdOSsfvMfNTnU35CFllHG2IFrxkEWOtgXljUXVznj5mdddHW5ZXuTNmazVTQpzfzqZIeW6C1LCW9b0Tlnnu6haW4HBRrR2zazOton4symbMQcJ50vbwmp1smpnBS+WX5KFmOqFqNVNCnO3HPvm04sX0i9dne/i0GDaRtg2EX82Wc5D/lFFO9iMyQmOyxeynZVVnViobt8unTxLDEitblKY20HKddPKds4vdZ3v4z3X4DyRha1j20T8GfUcaST5xyq6gm1Bexns5RmW9NyIjjku1JiNF7wwt0OU6/bUAHV1QP2FQpnLBjCtECcmOCfbRMp+s0kWpRxkjra9A8Hr5vH8B62q8oXhBLlrAinM7WZIuW5aLdl1UaUtqVyIPIFhJjxk2W9WbVRW8tCYu50Vh/EiWTYl8fcFojA3BDgp172BvkUc71ll4M0qzv/f8xMvSNlvNlFLzne+qF7Yqxe+GZaYilA9dDLoQcp1b9yLzTjYDEcc/Y8JZJtY2W9GcfkenLEfAVBz9rrVW7vlb7lzn0VdgoW5EVaumzVx+Ot8E+oz1zvlSNTZNsmy30ChOLR4AZQn+Pec1eYWkMLcZEW8VwlC/GDmr/NNZkiacOiRZWW/4YG8oATE339sdmNWYMe63ZC1umVhbkSW66YlvIFN1Pn+RmfpWhy7xeU2kbLf4jbmXhZo69y0o2f21Udx4+a29mD3YJYtIlLN7SJzpLVrnyuyG/g7ixIkKZe/7TDNDyKmJl+UVxBesnNQpJKxJyaYjYduKLZ6Bx6yOKF7UImsbhdZnWCzEgxlYSMTl+Unvcr7ojrm7nWQ5anjWkyFqkamZ7YQU5MlMj5xJaRUwHq9rpNGkvhZf9MGzlkpomPuvS62gX9iBJTrst+mL6DGZTuYiPYoZIsSJCl3d25lAZIQd1WwKjYRw3Iw9kKnjCSJhTQeTfvIsFRlRznl3xmhOFCRVwxctzbOM7GW5VNkYtJNINRGK2Rkqk0mlNrHjPjVSJxVmxDZmY5n/wMJg46HDFXFoiwbhox9+l6QzRbIVMuJICumbJXmgmwxYyt5IMjzOPOUrQIlABNBdl+yKKGGAIy5725LTosTYUU2c1PKFUtp4YHESXmvqVM6084XU/PxB5pSpeAKfnvSLs62OwzOxc7N30qR1T9Y+/72cxgc31/Hqbiq4q2ZL+XD+8fgSOkP70tfVNXxNb/x1sP7y+A03vAvdeKAR6UlA5zKuWq8DBidxm32CTGdCFIdaF4YfnvcE1Kn7MFivfOzEyaiwmMZNrH8ttWZhwKqf6f5wnT+/bYS0Hmc3uPsd7MU0DH33qnGfqebK6Dcw11XKHcPQWkJkhB3XzJN+UdQAeWY+yMdAk5bu2dsqWYCM0KIYtrKsA1bqZnIBGKEGFVMWxi2IaLmVinKBFxAXK/PvZpjiTBbFlWCqzp0Cw+6Wl3fcvVlqdPcGsWLV40bOw4kSlxcr5YwnJYg6y5OJGCiShNcTFuJUQgEugCV5NpRK4wlatUPjTJbzlBYLd1urTk3VF/6uC1BexpjxNyWJbgB0lZYTsoI6AKBSnq0tT9KIDoZCVHXk+fL4p6qW0WFZmteVFFBFRJcohEDLCdlELpAiFTSo+2gOiEjAXYBOLRqdflqlnVPGONuOdduffqyuc0rUVTXaFbxoonVUsPqXVTXNluGm80folpV39RZxYCBuZ72jtkydF3ihDqljd6WhsOpmXVUT+6jcAxZCCQpxbq86sFgartlEwZW01KoOJ7sgII4SRvnzmLUS7iV22lDGiJCyzrXPqQlG2Ck41/9FHQaWloH/vJDu0l39LbquItWp2/rGxDeITqNtg4ryayGBsoMhbRVnBPXj1st+XetMxCEHXBNZmanjzijnovq04Zrj/OqX16btJrpRQPmTWEVg7BXVUXdKA0F47yDTp1rIFhyJyVEwYU6/Er+9adWgnHeVt4dIAwZhhNoZiBD5qL2XbdoxN+mRstRz8X0pGurQQ/XJr/STU0gEEM/o15Wwjh30RsjInIFL+N8zSmZhwZeKz4TVRUghoy2kpVmIeM43X9s/nB9iupwbds7dtroW1uHU/MBp/I1R6ehJzuWAqd6V/Ve3gG4IfwFGE7nLzdU/yLvJGQl19w/bmJ68rYO4AN2vOGupeGRW/7x90FWNG8CHsOSrmjezEiAoDq5n5vNV0PWlVSjvpHbPz2aeoHT9og987fu9udNH+C+43YmIcoNJ7D4yf2Ef3pbY8hMxnHSK5nUgV7pjJCbOqk0VsydqKQxVHZ5MzsZ8U7HfFAHNq5D9USmB2msyEX3yFW8hs8aA0EuEwjZMgyPgxwBFyExbWOKkcIqhFRD5u74DcOdjGXI3NbE3HBqq1Gumj/4mRF3gsxYs0XuiIw1G8R3G8aW3EZxVWWqYmOzCPhtGGTatOdV8y+IjQ3Pz8dxsk3gt2GwKYbNeV4t/0IUGxvDtflAkE3Gdt9gqqlOup6xRf5+K/iAqIN5U6MU8SUZvl6pVOKeOy+jDmBjaxp9ouKMBCKV04aqHBas4yVKU7wEkkGgAqoIpAhPMoip0EwjHrhXxLaqmkJUlNG4pURiHBBxTjIIoRDhAwAiQpMMYiqt0ohH7gWgTwpRYcbiltpw2iEE+Jl0X+ua1WBPwVvWtDSbV1bMm6qlEBnnBmoBMxDUElZUnJgsNfmCy0n6ThacTxvQdOAVTUR5HmfU4X2zlz/gy4SO+ADJHcTF9byp8p4hGD88MLSBgcTjvAZWsLcRLWFFxZnJaUO13wETfEmQ28VEZSLGE0GiuIP3WSPKUc9PWyHch9O2Rh3M17QTh8Rz3px0QWTUG+BNa/gzFlrrvqUVEZ94TCY7TdAhyO1qSlFhRm8Ul4lZyAFURPk4/+lRD4PWtYGonXRHPcxFEgZC6wvSih2BuQkmxGYM2C4qKugM3Nkkk+4yiGCi+wqO+3K+lAgrijARlGajdXnuZ9K9JqPbs80dQ5+ylS/s/B1vz/S+wMRePCbjHNpd1S/9gJ+5rm0+xcCB0uJ/u3Q5Z430K4EfMBZanwcrDnoKYKfRgaV6++pBufzsEgs7H0jFjeS0yFl5OW+yxbQvDtWDedO3mLRi4syEtLd2Qtu32LUY9TFs+6S9mYmENDWp+0fxQGsGP3ITRmvTRj1hRcWpibBvumSj7+sfvSYDN31bDG8amggShIVMCIUiengvHLFm0ausmBSgTQpx9UioejcjUtGBOPE6dp8yJy0SUqEAHsItEMcQPB+AP3DihF2haJNCPBIJqeHRD1RUoOBoG3tPLiYWxELSgw9GAh3fB4iYHYj0QwxSSiEeiITU8OizLRWpeZyIdVxbyAnLJdgp3POt2zDYn/6mZhl2CnetDVablF1YbnDtDQZ5vrIfT0pppJtHNrUjk/d8QNq95hoSxHR0hNylyXGsxE++6QsGv1FKx0DlPR9wBfILLp7HabeEKZV4ZZ3g6aaUCX23GPyyaYKcyQffB1yBMzHunwhin7lLlgkRb1Q7bYiLOKR/+gn/uOO9ZTGwz43oSMz8d6CJ7tJIBKkhXF73afv6pgTScmLdSBI++GyY1LkGKJ6qp6EvTMgLM5N3oPmJEQQ9t64BYOPTjlHmE+Koii0acydBq8MpDsRoPSYZg0SjNAGoh1BMAVjh+9yQDKHzaDM/mQgjVIG1C9bcamsO4m9o7ao0EdiYVwyECe4exAMVO2jHqETKzdpuY7y3rs2/ADNEhtzpxu5/s1o/EAa9riXzOJ3OFOI9RlUPpFVmUYY+GdoWEeFBbj9re1/QGxHj7MSwOiCNDL3/zfIDCQOMK/NnvJMepUZV9UHvMtMoIIOANcYUN5rFNDNOphxV3cU2uDxCDEDuA2lkSOvfBsJoh01m2togGtoHJw4knBCsAuFATaQHCKRtkth/3rMxeCcHkEaGtP5NHKGXZxZp48vAi7wrzc+kywEBHJHsxcm45TyU6fihHmn9m0DCNE4tXLAW/bBT8xieOS8+af2bQMIg7/RNX7AGwc2RGSYGdQSu7Nun/8kZ03ve0sj4vq/5IfKwoEInxdscZeS2TN7qQFRP2rdVtW8dTYCHRQ2nVlNxQi5Dv735xYbTzqTb9i3mzCYccc+bB/2Xkda/tVntMdk2FFo8aU8/vDnqkSOs2Wv0LjPck/pb0j+q2k/RP1pTD6J/1AdhpRtx/ZswMmt404SJIMQipLgbLUYqwj2cos+Zbo18P/KN4ZtKs9ItGYgj20yZExbeY3qI4ERFqAeoe5z51sj3A98YuqlUK90GAskuk4uJBfERbuBERLgjqnudmYN8/w7fGLqpVCvdrNcPeOhTWINJG/ToHy3w9TxO7kF6fAKJSI9PRBwLA976AUhS7h6CggokIV71DSSHN9MSKCeC7PagvMl+v5+a/ykCZPUa+52wmfkHEiA7KPY7qAQU/vbirHFj+ss3hiiiJqfFemeqEFK12po1bgz3/DGrhRRe4HYDo29J86jngiig8zgR8ea58mXjDSa/Jf39x7HNaeMJ45nffIO5kdL39zeZsMLs5uHQ9yVz4/vDB+zxDMUVoqueWfvMzbVn1PkgCi2sJ79V99ntQ4bm9g2lJjVqcek8TnZkh7Tn2jKLhWvL9FKJFoaLDtT2Etv3ZxZ0eFFUAaQ4ijrwqIwi9nwzTYVfeIymDmMoV/hNI00qWBQOtRawaDoq09HEK1QNi6/012IEi7Ch42EjgiWo2p88FTCIuy/pB/yjcjkcIv9YRWHbE1cCpopk26MCARMeoQeRcDTibu6Kh/TWIWfL1iUPCdZTzjZdZzwEi2HI1cJhgVykPorb5swztiiGGR/B6bpqbyLGjCKtp8hJcBrys2DYQW6CyM2y+CCER5lSAfKVzOPiV2WGnKUX5X0u/hTegukm6pR1O5LNmaAue5wG0etEUdxx9dF2cVv1KAk1pib1mBJ6lJQ9XEI8uh4ZqUdK1yMg9ajoelSuW8y1PQJdD9c11CMl9cjoeniuNb8snThad7kNYniJZZ6rSBuNcRRF5B45oYYyqYci9Mgpe0SEjOl6BKQeLl2PlNSjpOtRknqk5vRwST0Cuh7jKIrM/2VRee56IeoBwZcFihHv3mVOFYOkNOc66xAkqQAk5SmgUIFzLGaAYh06x2QJJ+ohOkea8yqv036UYwcJhxmnKsftx2jlILiYQYlsGDpJFUMJ10UnwW4PRtQqdJZahSCimKKzoNeBEF4HnQZdl0dNi/ZCxqHzYKfHoVy3rajyAB0Iu13Y4G0CdCTsbVK4UHc7IToU9vNRHybUE9VDdCzE1UYVKTQIpvlwGaJlbPRjj6I458QolbM/60j7t6N17PU/vJQT0+mm7I9mLMdaApJxiW0ge7nqeCApT8HU8V4duL1SYGZ/80IC1XhaFDeNTEc37Shbo1U5huFUGTxztHaFxna2x76tdHkhWfb2N1q9QI2NQLJo1dpiCdcZ7hsHsI9kbZwvDmaFWO+RC3qhcdPqWR775ZnQOF9wjdYl4udT07q+cLluJkRcUZLBLuOeahRFavbNTeVZFoQKKMEKSZQv4blwGpvmeM+ZfMa2s0bKG9MN5KIosiFsdOdSqzD/5iwbwuxPAYPUi/3/YzOzg1kmhDkHIhoosJkVfeftbyzPMiGMPWl/GRCd9kSj9dGFHRUpOV/UP1QUD3YzIeZXsGJ40wO50ClWZPUiAyJCsyaCuCubHziNny9SA0dv+9q3ikwIG/6kf+V4Dssz2fd21ZQrzEFSoHiFZ1Tqg8c4lqC1Wz+0+JQZ7cffvy4UqtZWYHK8wYNvNPt17gtI+KjU8wVejEj42oUdzKSR432hcCguaEjLeMF8AQkflbpvPzJz19WOWruJhs+ofMEZ1Y/34tPENzECt9YlVjuivnBEjEplma37bWs3XsjxqnSgFupsANOfOLGDGQ549fnC0R06KvV8YTgdXH7n7sGMjleFUdfCF+USzhfbCPX9FRTZpqNSCfH3dw9mfLwq43f37Bfe/NP/30uJ/HP78YmI9fGY6qZt46fYER2VSsIb7gLg41XZTBUNTI/3fDPbz2HjLK/5bM+hy1THe3BK9P2LeEFHpbZ2y99yZ1SLuoOPV3VjW59Ck5tWn7lPhSZvFcXtRb1gvu3J9nNYHtaSHUounOmo1POF6HtyLx6j/fIE84LjPZkhCMvyjPq2KQm7w1bekatyVKrsfLcbTwsHM8JO+8c4+PrBNzqV/fIMFJIDN/dtVwasDjtduQmabMV0lYUkSQgFjFcz/ceTwI0QYwxEYFNqVyCwne3zDhCRj4SomogSgfRqVySuwTabIUQdjHoaQyIwzjuJ8mbUtjysh8vVl6WNxDaroe92AYaYRP4awKjvFQooRFRQkqoGRshggJhtw6UnScQJ81dnIdWokDug0ymgJFWNUeAWyGQYL1GaCnknEhUnzF9BSKhRIaFHVFCSq6oCuEAWQ4XmQnY97WUjs9Uorzv05L7Z/GFD+3O3B0y6pw2htmxu9uQLeRuDHJZL3n99x+Yvqw6sDZJO/C9eBHFHVd/UWcWQwdDvuOXNzG7lRME6Ir4HVlOqZuZDuLhlpWTJ/J7soEEM2QrBHVVrOpzaHzX0Xm4JfacwXC1hNYzv/qjnUfPTaWhpHQU/ITl6W3XcBavTt7XCsE4Ww/hDTfgQOMB2UMGFekeEdkc9al5GPYXMwrUXzVXXZ/q/1UwvmmPeFFYxPKegIi5Lk8kfSWhrdKmoQ9ifNgZWE4kABrtTG3veZgbOGd2kObPaak62iJyj5EbSdQ1kmRpJIDyvAvYypWTUSxq4SAa1LUDLUa+Px2tbDXr4j690GVf25QqpkfSMmBi7dJ7ISAzVHWt+QMVRuBjkqb6ZtsqpHP1HYuFVa6tMFnyM89fICnxWz6ozqZGUWkP+ZGDgUvukyXA6f7mh+hfYAbCKmDcV1HgT7UNT8zGc3spruEc19GTHns0QoTvlaYPknzfyjsrZeVMx4NWjZo6BmdmTHV+4a2mwX026Mu3IIDD3IzG4nnpPp73909saXXy3lantmD35a5f16hu5/dOjqZdx3iF+t+72580s50WdZIjkQmWIxACtosfmleduUQxtiuUGVtOWpWgFMM+PRnAMZEgHUQjCCtq5bODJ2+q/9O7mmCfNJJCoeiKOxImkaAUww5sBFeXgN6YgrLBdiNo3cdlTJq2ny8G+Ii9jEkvpaycDQGH+VOFKitzlqL5MKFtN7bOb2VyocoemuPSt3NZCG5biGlucBmt4tmtsGe/zJXj7JAhf8Y1gnO2NH4jGeTGFFtPDaDftiJWxtgkBOdc6+wlqWI1Mj37U4uqkfvwbbQnXb4ksApfDb2S4b0cZHgbTNgfa3rR6keVRnxWHVveQPJh0I9ZevMjyiBf4xFxOcWgrMDvK8N6We4AXd+AuskeO9yxeZIPQDKLYt2+1ejZIsW/bbt0JN63u1pWxod+vtAUhtNLlG91zm+GJpAez5yguUzweGc0RoraY4QnrBUxUfP/CKoM/nShg+0Pbfk7Cu3PnTNv3jWbI/NlJh+a0+8avb18fB4IwZe60PX6hfN/wLE1/yPh5nE3m1GvkbB1nGxEQvtzTvMfwlPEh22e8a5nRT/OhrdlrpQC4+zVfSsOrFZuYvjZIz6Q1ndL3X1txP/d0fKlZP9sWmVdyaxbvwPnq61ud5t9c7ZKJLc/VJzjfTx9svvU8JbRMP/+BhbUaP0QrmFnzRsr1wqM0v7WBwIoNbnWlmMSyWknzh2hrvTHX8x73bSFgJ86WF1DV1jRca1NWMrGzrdmc67mHthCi6iv3c7NbD7ew1HQi9WemtqQWYfNy85nZ83rA87pnmx2E6J6rh9O8qZEVU6XcMDyh3TCcCN3xaHmLvaKfnr4pqZmN3LkMz3v51zbRt1t3G6ovrZawUU9Z2Wm6DcMTSbthOFhRW5hJURY7oJ+OvimpWWIfN1Ke9/i+DcDh8kkPq/JXaFajBfrpNgzXl9EhPCYFK2rLzNFZCqCBBtXe5TKBA7ALRIjTyWqWV+mjTOg2DCfsF07eMBwP86eP7juNoXu+SkOYfzw3Wwa/rfiolO94W2NkDy+n27tcJjCAZZ3ibz2FbsPwH+qrtNgwHFnN3WncnYIM0hcZsnj41KhHH15OtXc5HLwKcOCVf7Z9UG0YTtov3CdsGC5xGvdyt9xf2GnDAQfdxKI9nLrd42PGH15OtXc5PJBPyDt9xKDaMJy0X7jfYsNwx7ypQnU4lcz/6f/+QZU8a5w//ZRq7/Lmp8DCnXRQbRiu3y+c2IpALsvQEuYfNsa5+M4w9DnRVHuXg8KX4QlOxN9yaDYMJ+wXbhp95T6E++uO4INm73JgMM5d3MaIsFyBZsNw/X7h5nHTS/d25uu+ITPxzdDsXS6TARjAc6P5w39QHd7bsdMGeV64T7NhOGG/cLOM884/qJ5M569O7a7qPfe7vmxCsXe5sDPQ6uMmpidvs+eF+zQbhhP3CzfnFXtk5DgXjylnDy+n2Ltc2BmpdXI/N5v/0w8reV54K7oBSeJ+4eYgj4wUzyN/WyN9eHmrvcs190HBICGOZBI7EEc6pe5/5dKnaxWEeqQyBoc3hR6T0bYHBgPinA9iB6oNwxNJ1SqAOtMgNaefCOLRS3SlB4LJVsswWnSg2zA8oWiViM2hGuRuumI0P6wUzy4HBRStDW0YbvMFa9BiQJjOBov0E3+Bke4RKzwaDAYDeML8E2sJK+afglBoMT3i6mv7bM+rguo9d1iffQE1SKTP/shkIJwGiWR/koGAGiTsz0C4tMns+W/Pf3v+ewSKu73MWykuAyTAjbjempqZHyFiyQJv9eGMjxDlI328GlkTrc5EoPk6jwaqST9G8MDVIEjS4pwt2kemHo0AakHKSGIMHK6WwI9E6s9uZcmVlhD8jzaDHoluN8e/P3fLmistIeDXizi5n+NDIBPYkUj9HrfRLYuutEQw6bIH55rmGjUgRyJJF0Byy4orLSE8msKXy+nDV9L5ePQs+CBcQUGDNVdagvwz/cJDKC0T2PFC9JivveSMF82/0hICphA6ztsg8sO/PA1JTLrSEjrzdd/uN17A6I4lV1oiu0sH1lHvDBR9im/JlZaoyu8MYTX1NAGjA1lzpSWl/I4b4/y0wb5BSQMMPWn/medu+aZdaQkld3gXXz/ws1+FoCft3aT9z5ElV1rCNLKky5sh6qT75N++/9K7//K0L6240hKmkSkNBAmRfc8mmVhxpSVMwzNkDn6w79mU2OxKS4DnU1j+aaCVTOBtFeVm809MVOFtaS/JPw21AsqTIOz5b89/e/7b8x/ZuAeSJqeLW6gdeN4FbvOK548l43nB9QYu+Xb/ceR5eOOhatx4xO0e3/Meb+CQl2/cm5Dv4Qn7NXfckoaTuQRyvjq+1rglXPGW9M/WaUPeh5dp3mTCDZHS9792M+R/WMXnyvclN8T3t2afvRuiAMD6E3e+9rzPC7n2zEfPB1EIIPbvHuHGzUMeyA2lJhk6VcdDDklaujyQtDmo6VjdCyA0pr9BnBbgqBwjyuOAo0VzkOBAAqPwGBkieh1QRDjGpg+KahVqv59NDYnvy/uE7zEAQ1mnJI40z4BQuvHIfzIWcZnB31d5o1y2nJYzPsomn7iAl1E+72xDN66UngfyrNIQYTJ7/tt7gOfaPSXMePzVQ3vnrgIa+3Y/cxLYsPf58CCHnU8dCjz0p2LXxE7nXAYhifQ1kQkkSqT/Sr4E2Kgn0ggngx9ANO5IFWkEVmLiktkRifRfiGqqg3lTxSeJPIJnE1JFGKGV8OLLZUUk0n/x+Hge0xi1XJY2xCeJPIJnE1KFGUUVE5eMITvild982DH+UHO4XOkLZh2NOunaqJcIVZaZtSGbiDowmzdVx7nNm07VvT/+lR2nl5vVhGoMmREvzD/gmHRrw+moN+m24ZhdnjbGeVsHLsKd1VKTphNM5JyKJK+p800cqv6hqsaQRVFTLTvuAOyif/whPBarYsirSiOBE3PaTCghW5pxoQwM3HrNbC5DXkuzjksLrAaRn9ghsFo8LiuqOrKnJvxFVSVicusWLwz6gg0zqBUNp5itDqc1F+uSiKdfjuRtuaZy5BBvf+hVVM2IXrS/c7/bhU77hJ82a+Ni86YSUiympVnF01JTIJLFcD57urypyQzvH5XvhxT9o85Udqam6h/NlAaCYFZLryTJ99NcSZKmTjJ7upLUap6bIPMbMke1+7NmgIOLbR2wkR/PDXvnDtAIUrtnB7zaOwg=) **Figure: Qualcomm virtualization solution** For more information on type 1 hypervisor, see [gunyah-hypervisor](https://github.com/quic/gunyah-hypervisor). ### Guest VM build support To compile the guest VM kernel and initrd images, use the following procedure: During the guest VM kernel compilation, both the guest kernel and the minimal `rootfs` are built. `qcom_vm_defconfig` serves as the base configuration for compiling the guest kernel. The `qcom_vm_debug.config` builds the debug variant of the guest VM. `crosvm` serves as the VMM and interacts with the Gunyah VM manager in the kernel through the `/dev/gunyah` interface to create VMs. **Build a guest VM** For more information on `crosvm` support, see [Secondary virtual machine](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-27/platform_software_features.html#sub$secondary_virtual_machine). After the build process completes, certain images are generated. These images get stored in the `/build-qcom-wayland/tmp-glibc/deploy/images//qcom-console-image` directory. To compile the guest VM, use the following command: export SHELL=/bin/bash && MACHINE=SOC DISTRO=qcom-wayland source setup-environment bitbake qcom-console-image Copy to clipboard **Guest VM initrd image** - The `svm-initramfs-qcom-image-.cpio.gz` file is created for the guest VM. This image contains an initrd image, used during the boot process. - The `svm-initramfs-qcom-image-.cpio.gz` file is generated in the `build-qcom-wayland/tmp-glibc/deploy/images//svm-initramfs-qcom-image` directory. - On the booted device, the image is stored in the `/var/gunyah/initrd.img` file within the system rootfs. **Guest VM kernel image** - `svm-initramfs-qcom-image-.cpio.gz` image is generated for the secondary virtual machine (SVM), which is the guest kernel image. - The SVM kernel image is generated in the `build-qcom-wayland/tmp-glibc/deploy/images//linux-svm-kernel-qcom` directory. - On the booted device, the image is in a `/var/gunyah/Image` file within the system rootfs. Both the `initrd.img` and the SVM kernel images are packed in the `system.img` file. The final `system.img` file is found in the same directory as the other images `/build-qcom-wayland/tmp-glibc/deploy/images//qcom-console-image`. ### Launch guest virtual machine Use the `crosvm` VMM and the Gunyah hypervisor to launch the guest virtual machines. Use the following command to launch the VM: # /usr/bin/crosvm --log-level=debug --no-syslog run --disable-sandbox --hypervisor \ gunyah --protected-vm-without-firmware \ --serial=type=stdout,hardware=virtio-console,console,stdin,num=1 \ --serial=type=stdout,hardware=serial,earlycon,num=1 \ --initrd /var/gunyah/initrd.img --no-balloon --no-rng \ --params "rw root=/dev/ram earlyprintk=serial panic=0" \ /var/gunyah/Image Copy to clipboard The following are the supported parameters for the `crosvm` command to launch guest VMs: Table: Supported parameters | Parameters | Description | | --- | --- | | /usr/bin/crosvm
Copy to clipboard | Shows the path of the `crosvm` binary | | --log-level=debug
Copy to clipboard | Sets the logging level to debug while running `crosvm` | | --no-syslog
Copy to clipboard | Disables syslog | | --disable-sandbox
Copy to clipboard | Disables sandboxing | | --hypervisor gunyah
Copy to clipboard | Specifies the hypervisor to be used (in this case, `gunyah`) | | --protected-vm-without-firmware
Copy to clipboard | Indicates that the VM is protected and does not require firmware | | --serial=...
Copy to clipboard | Configures serial devices for input/output | | --initrd /var/gunyah/initrd.img
Copy to clipboard | Specifies the initial RAM disk (initrd) image | | --no-balloon
Copy to clipboard | Disables memory ballooning | | --no-rng
Copy to clipboard | Disables the entropy source used to seed the guest OS entropy pool | | --params "..."
Copy to clipboard | Specifies the kernel command line options | | /var/gunyah/Image
Copy to clipboard | Specifies the path to the kernel image | **View VM and boot logs** The following sample output shows the VM logs: bash-5.1# cat /proc/cpuinfo processor : 0 BogoMIPS : 38.40 Features : fp asimd evtstrm aes pmull sha1 crc32 atomics cpuid asimdrdm dcpop asimddp CPU implementer : 0x00 CPU architecture: 8 CPU variant : 0x0 CPU part : 0x048 CPU revision : 0 bash-5.1# cat /proc/meminfo MemTotal: 161656 kB MemFree: 124940 kB MemAvailable: 124032 kB Buffers: 0 kB Cached: 19816 kB SwapCached: 0 kB Active: 12 kB Copy to clipboard The following code snippet shows the boot logs: [2022-04-28T17:43:03.290477745+00:00 DEBUG crosvm::crosvm::sys::linux] creating hypervisor: Gunyah { device: None } [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0480] [ 0.000000] Linux version 6.6.0 (oe-user@oe-host) (aarch64-qcom-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Wed Feb 7 04:56:59 UTC 2024 [ 0.000000] KASLR enabled [ 0.000000] random: crng init done [ 0.000000] Machine model: linux,dummy-virt [ 0.000000] earlycon: uart8250 at MMIO 0x00000000000003f8 (options '') [ 0.000000] printk: bootconsole [uart8250] enabled [ 0.000000] efi: UEFI not found. [ 0.000000] software IO TLB: Reserved memory: created restricted DMA pool at 0x0000000090000000, size 64 MiB [ 0.000000] OF: reserved mem: initialized node restricted_dma_reserved@90000000, compatible id restricted-dma-pool [ 0.000000] OF: reserved mem: 0x0000000090000000..0x0000000093ffffff (65536 KiB) map non-reusable restricted_dma_reserved@90000000 [ 0.000000] NUMA: No NUMA configuration found [ 0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x0000000093ffffff] [ 0.000000] NUMA: NODE_DATA [mem 0x8ff509c0-0x8ff52fff] [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000080000000-0x0000000093ffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000080000000-0x0000000093ffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x0000000093ffffff] [ 0.000000] On node 0, zone DMA: 16384 pages in unavailable ranges [ 0.000000] cma: Reserved 32 MiB at 0x000000008d600000 on node -1 [ 0.000000] psci: probing for conduit method from DT. [ 0.000000] psci: PSCIv1.1 detected in firmware. [ 0.000000] psci: Using standard PSCI v0.2 function IDs [ 0.000000] psci: MIGRATE_INFO_TYPE not supported. [ 0.000000] psci: SMC Calling Convention v1.3 [ 0.000000] percpu: Embedded 31 pages/cpu s86568 r8192 d32216 u126976 [ 0.000000] Detected PIPT I-cache on CPU0 [ 0.000000] CPU features: detected: GIC system register CPU interface [ 0.000000] CPU features: detected: Spectre-v4 [ 0.000000] CPU features: kernel page table isolation forced ON by KASLR [ 0.000000] CPU features: detected: Kernel page table isolation (KPTI) [ 0.000000] alternatives: applying boot alternatives [ 0.000000] Kernel command line: panic=-1 console=hvc0 earlycon=uart8250,mmio,0x3f8 rw root=/dev/ram earlyprintk=serial panic=0 [ 0.000000] Unknown kernel command line parameters "earlyprintk=serial", will be passed to user space. [ 0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes, linear) [ 0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes, linear) [ 0.000000] Fallback order for Node 0: 0 [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 80640 [ 0.000000] Policy zone: DMA [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.000000] software IO TLB: area num 1. [ 0.000000] software IO TLB: mapped [mem 0x0000000089600000-0x000000008d600000] (64MB) [ 0.000000] Memory: 116344K/327680K available (14528K kernel code, 3938K rwdata, 7112K rodata, 6080K init, 571K bss, 178568K reserved, 32768K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] trace event string verifier disabled [ 0.000000] rcu: Preemptible hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=1. [ 0.000000] Trampoline variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] GICv3: 988 SPIs implemented [ 0.000000] GICv3: 0 Extended SPIs implemented [ 0.000000] Root IRQ handler: gic_handle_irq [ 0.000000] GICv3: GICv3 features: 16 PPIs [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x000000003ffd0000 [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. [ 0.000000] arch_timer: cp15 timer(s) running at 19.20MHz (virt). [ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns [ 0.000000] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 4398046511078ns [ 0.002616] arm-pv: using stolen time PV [ 0.003945] Console: colour dummy device 80x25 [ 0.005300] Calibrating delay loop (skipped), value calculated using timer frequency.. 38.40 BogoMIPS (lpj=76800) [ 0.008475] pid_max: default: 32768 minimum: 301 [ 0.009923] LSM: initializing lsm=capability,selinux,integrity [ 0.011702] SELinux: Initializing. [ 0.012884] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes, linear) [ 0.015180] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes, linear) [ 0.017809] cacheinfo: Unable to detect cache hierarchy for CPU 0 [ 0.019820] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. [ 0.022107] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1. [ 0.024472] rcu: Hierarchical SRCU implementation. [ 0.025891] rcu: Max phase no-delay instances is 1000. [ 0.027767] EFI services will not be available. [ 0.029190] smp: Bringing up secondary CPUs ... [ 0.030713] smp: Brought up 1 node, 1 CPU [ 0.032002] SMP: Total of 1 processors activated. [ 0.033471] CPU features: detected: 32-bit EL0 Support [ 0.035048] CPU features: detected: Data cache clean to the PoU not required for I/D coherence [ 0.037742] CPU features: detected: Common not Private translations [ 0.039678] CPU features: detected: CRC32 instructions [ 0.041275] CPU features: detected: LSE atomic instructions [ 0.043105] CPU features: detected: Privileged Access Never [ 0.044891] CPU: All CPU(s) started at EL1 [ 0.046161] alternatives: applying system-wide alternatives [ 0.050071] devtmpfs: initialized [ 0.051500] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.054767] futex hash table entries: 256 (order: 2, 16384 bytes, linear) [ 0.057122] pinctrl core: initialized pinctrl subsystem [ 0.058872] DMI not present or invalid. [ 0.060334] NET: Registered PF_NETLINK/PF_ROUTE protocol family [ 0.062534] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations [ 0.064964] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.067353] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.069943] audit: initializing netlink subsys (disabled) [ 0.071632] audit: type=2000 audit(0.056:1): state=initialized audit_enabled=0 res=1 [ 0.074141] thermal_sys: Registered thermal governor 'step_wise' [ 0.074143] thermal_sys: Registered thermal governor 'power_allocator' [ 0.076118] cpuidle: using governor menu [ 0.079536] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers. [ 0.081693] ASID allocator initialised with 32768 entries [ 0.083662] Serial: AMBA PL011 UART driver [ 0.085582] Modules: 2G module region forced by RANDOMIZE_MODULE_REGION_FULL [ 0.087812] Modules: 0 pages in range for non-PLT usage [ 0.087813] Modules: 516176 pages in range for PLT usage [ 0.091914] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages [ 0.099837] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page [ 0.101815] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages [ 0.107895] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page [ 0.109875] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages [ 0.116064] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page [ 0.118084] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages [ 0.124190] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page [ 0.128423] ACPI: Interpreter disabled. [ 0.132206] iommu: Default domain type: Translated [ 0.133734] iommu: DMA domain TLB invalidation policy: strict mode [ 0.135749] SCSI subsystem initialized [ 0.140957] usbcore: registered new interface driver usbfs [ 0.142744] usbcore: registered new interface driver hub [ 0.144353] usbcore: registered new device driver usb [ 0.149976] pps_core: LinuxPPS API ver. 1 registered [ 0.151488] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.162194] PTP clock support registered [ 0.163436] EDAC MC: Ver: 3.0.0 [ 0.164532] scmi_core: SCMI protocol bus registered [ 0.166176] FPGA manager framework [ 0.171350] Advanced Linux Sound Architecture Driver Initialized. [ 0.173213] vgaarb: loaded [ 0.174176] clocksource: Switched to clocksource arch_sys_counter [ 0.235827] VFS: Disk quotas dquot_6.6.0 [ 0.237108] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.239428] pnp: PnP ACPI: disabled [ 0.241483] NET: Registered PF_INET protocol family [ 0.243293] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear) [ 0.245817] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.248604] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) [ 0.251034] TCP established hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.253323] TCP bind hash table entries: 4096 (order: 5, 131072 bytes, linear) [ 0.255664] TCP: Hash tables configured (established 4096 bind 4096) [ 0.257584] UDP hash table entries: 256 (order: 1, 8192 bytes, linear) [ 0.259650] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear) [ 0.261768] NET: Registered PF_UNIX/PF_LOCAL protocol family [ 0.263829] RPC: Registered named UNIX socket transport module. [ 0.265673] RPC: Registered udp transport module. [ 0.267186] RPC: Registered tcp transport module. [ 0.268641] RPC: Registered tcp-with-tls transport module. [ 0.270380] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.272421] PCI: CLS 0 bytes, default 64 [ 0.273680] Unpacking initramfs... [ 0.278241] kvm [1]: HYP mode not available [ 0.279783] Initialise system trusted keyrings [ 0.286209] workingset: timestamp_bits=42 max_order=16 bucket_order=0 [ 0.288474] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 0.294313] NFS: Registering the id_resolver key type [ 0.295881] Key type id_resolver registered [ 0.297197] Key type id_legacy registered [ 0.302225] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 0.304407] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering... [ 0.310289] 9p: Installing v9fs 9p2000 file system support [ 0.320607] NET: Registered PF_ALG protocol family [ 0.322022] Key type asymmetric registered [ 0.330205] Asymmetric key parser 'x509' registered [ 0.331833] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245) [ 0.334140] io scheduler mq-deadline registered [ 0.342201] io scheduler kyber registered [ 0.343459] io scheduler bfq registered [ 0.353648] pci-host-generic 10000.pci: assigned reserved memory node restricted_dma_reserved@90000000 [ 0.362220] pci-host-generic 10000.pci: host bridge /pci ranges: [ 0.364055] pci-host-generic 10000.pci: MEM 0x0002000000..0x0003ffffff -> 0x0002000000 [ 0.370193] pci-host-generic 10000.pci: MEM 0x0094800000..0xffffffffff -> 0x0094800000 [ 0.372927] pci-host-generic 10000.pci: Memory resource size exceeds max for 32 bits [ 0.382221] PCI: OF: PROBE_ONLY enabled [ 0.383535] pci-host-generic 10000.pci: ECAM at [mem 0x00010000-0x0100ffff] for [bus 00] [ 0.386134] pci-host-generic 10000.pci: PCI host bridge to bus 0000:00 [ 0.394279] pci_bus 0000:00: root bus resource [bus 00] [ 0.395912] pci_bus 0000:00: root bus resource [mem 0x02000000-0x03ffffff] [ 0.397972] pci_bus 0000:00: root bus resource [mem 0x94800000-0xffffffffff] [ 0.406329] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.409065] pci 0000:00:01.0: [1af4:1043] type 00 class 0x00ff00 [ 0.419597] pci 0000:00:01.0: reg 0x10: [mem 0x02000000-0x02007fff] [ 0.425686] Freeing initrd memory: 6464K [ 0.428105] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 0.430575] pci 0000:00:02.0: [1b36:0011] type 00 class 0xffff00 [ 0.432661] pci 0000:00:02.0: reg 0x10: [mem 0x02008000-0x0200800f] [ 0.436076] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.438471] virtio-pci 0000:00:01.0: assigned reserved memory node restricted_dma_reserved@90000000 [ 0.443123] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.445534] 2e8.U6_16550A: ttyS0 at MMIO 0x2e8 (irq = 14, base_baud = 115200) is a 16550A [ 0.448422] 2f8.U6_16550A: ttyS1 at MMIO 0x2f8 (irq = 14, base_baud = 115200) is a 16550A [ 0.451159] 3e8.U6_16550A: ttyS2 at MMIO 0x3e8 (irq = 15, base_baud = 115200) is a 16550A [ 0.453983] 3f8.U6_16550A: ttyS3 at MMIO 0x3f8 (irq = 15, base_baud = 115200) is a 16550A [ 0.456935] msm_serial: driver initialized [ 0.460508] printk: console [hvc0] enabled [ 0.461772] printk: bootconsole [uart8250] disabled [ 0.475305] loop: module loaded [ 0.475957] megasas: 07.725.01.00-rc1 [ 0.476374] tun: Universal TUN/TAP device driver, 1.6 [ 0.477187] VFIO - User Level meta-driver version: 0.3 [ 0.477691] usbcore: registered new interface driver usb-storage [ 0.478075] i2c_dev: i2c /dev entries driver [ 0.478575] sdhci: Secure Digital Host Controller Interface driver [ 0.479311] sdhci: Copyright(c) Pierre Ossman [ 0.479612] sdhci-pltfm: SDHCI platform and OF driver helper [ 0.479932] ledtrig-cpu: registered to indicate activity on CPUs [ 0.480343] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping .... [ 0.480864] usbcore: registered new interface driver usbhid [ 0.481329] usbhid: USB HID core driver [ 0.482042] NET: Registered PF_PACKET protocol family [ 0.482342] 9pnet: Installing 9P2000 support [ 0.482635] Key type dns_resolver registered [ 0.484383] registered taskstats version 1 [ 0.484775] Loading compiled-in X.509 certificates [ 0.486896] page_owner is disabled [ 0.487629] Key type .fscrypt registered [ 0.487943] Key type fscrypt-provisioning registered [ 0.488391] clk: Disabling unused clocks [ 0.488759] ALSA device list: [ 0.489050] No soundcards found. [ 0.490311] Freeing unused kernel memory: 6080K [ 0.498264] Run /init as init process Starting version 250.5+ bash: cannot set terminal process group (-1): Inappropriate ioctl for device bash: no job control in this shell bash-5.1# Copy to clipboard **Launch a VM with two CPUs and default RAM** To launch a guest VM with two CPUs, use the `--cpus num-cores=2` parameter: #/usr/bin/crosvm --log-level=debug --no-syslog run --disable-sandbox --hypervisor \ gunyah --protected-vm-without-firmware \ --cpus num-cores=2 \ --serial=type=stdout,hardware=virtio-console,console,stdin,num=1 \ --serial=type=stdout,hardware=serial,earlycon,num=1 \ --initrd /var/gunyah/initrd.img --no-balloon --no-rng \ --params "rw root=/dev/ram earlyprintk=serial panic=0" \ /var/gunyah/Image Copy to clipboard The following code snippet shows the VM logs: bash-5.1# cat /proc/cpuinfo processor : 0 BogoMIPS : 38.40 Features : fp asimd evtstrm aes pmull sha1 crc32 atomics cpuid asimdrdm dcpop asimddp CPU implementer : 0x00 CPU architecture: 8 CPU variant : 0x0 CPU part : 0x048 CPU revision : 0 processor : 1 BogoMIPS : 38.40 Features : fp asimd evtstrm aes pmull sha1 crc32 atomics cpuid asimdrdm dcpop asimddp CPU implementer : 0x00 CPU architecture: 8 CPU variant : 0x0 CPU part : 0x048 CPU revision : 0 bash-5.1# cat /proc/meminfo MemTotal: 161656 kB MemFree: 124940 kB MemAvailable: 124032 kB Buffers: 0 kB Cached: 19816 kB SwapCached: 0 kB Active: 12 kB Copy to clipboard **Launch a VM with two CPUs and 512 MB RAM** To launch a guest VM with two CPUs and 512 MB RAM, use the `--cpus num-cores=2 --mem size=512` parameters. #/usr/bin/crosvm --log-level=debug --no-syslog run --disable-sandbox --hypervisor \ gunyah --protected-vm-without-firmware \ --cpus num-cores=2 --mem size=512 \ --serial=type=stdout,hardware=virtio-console,console,stdin,num=1 \ --serial=type=stdout,hardware=serial,earlycon,num=1 \ --initrd /var/gunyah/initrd.img --no-balloon --no-rng \ --params "rw root=/dev/ram earlyprintk=serial panic=0" \ /var/gunyah/Image Copy to clipboard The following code snippet shows the VM logs: bash-5.1# cat /proc/cpuinfo processor : 0 BogoMIPS : 38.40 Features : fp asimd evtstrm aes pmull sha1 crc32 atomics cpuid asimdrdm dcpop asimddp CPU implementer : 0x00 CPU architecture: 8 CPU variant : 0x0 CPU part : 0x048 CPU revision : 0 processor : 1 BogoMIPS : 38.40 Features : fp asimd evtstrm aes pmull sha1 crc32 atomics cpuid asimdrdm dcpop asimddp CPU implementer : 0x00 CPU architecture: 8 CPU variant : 0x0 CPU part : 0x048 CPU revision : 0 bash-5.1# cat /proc/meminfo MemTotal: 418304 kB MemFree: 378356 kB MemAvailable: 376048 kB Buffers: 0 kB Cached: 20308 kB SwapCached: 0 kB Active: 0 kB Copy to clipboard **Launch a VM with block device** To launch the VM, use the following command: #/usr/bin/crosvm --log-level=debug --no-syslog run --disable-sandbox --hypervisor \ gunyah --protected-vm-without-firmware \ --block disk.img \ --cpus num-cores=2 --mem size=256 \ --serial=type=stdout,hardware=virtio-console,console,stdin,num=1 \ --serial=type=stdout,hardware=serial,earlycon,num=1 \ --initrd /var/gunyah/initrd.img --no-balloon --no-rng \ --params "rw root=/dev/ram earlyprintk=serial panic=0" \ /var/gunyah/Image Copy to clipboard - Use the `--block` flag to specify the block device name (in this case, disk.img). - This flag creates `/dev/vda, /dev/vdb`, and `/dev/vd*` nodes in the guest VM. bash-5.1# ls -al /dev/block/ total 0 drwxr-xr-x 2 root root 220 Jan 1 00:00 . drwxr-xr-x 9 root root 2980 Jan 1 00:00 .. lrwxrwxrwx 1 root root 6 Jan 1 00:00 254:0 -> ../vda lrwxrwxrwx 1 root root 8 Jan 1 00:00 7:0 -> ../loop0 lrwxrwxrwx 1 root root 8 Jan 1 00:00 7:1 -> ../loop1 Copy to clipboard To mount the device in the guest VM, run the following command: `mount -t ext4 /dev/vda /mount-point`. bash-5.1# mkdir /data bash-5.1# mount -t ext4 /dev/vda /data [ 112.311376] EXT4-fs (vda): mounted filesystem 4b577765-65ad-4fd8-910c-0eac437d8afb r/w with ordered data mode. Quota mode: none. Copy to clipboard ### KVM overview A kernel-based virtual machine (KVM) is an open-source virtualization module integrated into the Linux kernel. This integration allows the KVM to act as a hypervisor. KVM facilitates hardware-assisted virtualization for guest operating systems. Note KVM is supported on QCS8275 and QCS9075 SoCs only. ![../_images/kvm-blockdiagram.png](data:image/png;base64,UklGRjZEAABXRUJQVlA4TClEAAAvDwO6AGaB5LaRJEm2///qjMiqiJrlGhETwAf1YUpJSGxVzy7uPfWmy0bkJUl3uGRS4s5mUvUNG3gujRAk6cGwKPoNKAc/Zd7RoUanrabUPUiilqht++qhrugw/2db+85V37e9wVYaM+OHBZ75pm3AqzhX/QcDybJ+Q4r9qZI9bVidsDbV5iSRAEkgae5XDATQie3EY8mFXE6t1Pv8v6s5csRAhmGHEzLskGGFE07YITFRhwwnnFBhhx1O2OGECglFChUqZCivDhkqa+Ce3+937j3unnNe9hRK3jAbXHnpJYYl79lS5G4klFxW8tIrSxCoUJ4gOv7J7BaqE5nbmbxdX8BbUYGYaH2tKRSIlaGyNSmjfdfWGmLlelDrBqh1jQbR6FK4YwiiUXix6QLyUmVacLlpRWtCmYprvbedrnfMK7oAG5g/oUBsRCzkam2tuRgHA/vnPNLsfQaLwWAwOFicp1QMFgeLi8XgYLF433c4uLi4GNonGAyeBQYAwHGk7N43GDwMBoODxWIxGBzcpx8Wi4PF4mIxWFy7YLB4/w2etf+yINkKWmnui0aj2aMVFITLZX/tRMA1x7LTEBrChdAQGkKHlz+U/3znSPrnlv+xvG8QTjyxF1UJhCiIwWQdd5URnMhLfAA0GyVKbvqiYfEAKH0E2rtnSR8Lh0qNRZG8m4GUd+Yh8UC4SvVA2AyMtI2cfQknYfAkDB58ggCSMcYYYYQQxhi//7Jo2woa5ziTFot44upbYEAxv3YjUHYkyRlzxpQ2ZUwZM9qMse/ny8w/dz27OwsOF8qFH1Eg6hELJvpMgB0IiCWBpFYytLWhLBgDjtiAAtERpV19A8BLb1AH1pIz4xKk5KuAOS2CGXDrCKLBSlhtdbCSiyCAZJxRwhhjjFFGNMg20iMdyuEMYyiH8kj9l0XbVlArm66lAS4YncfhgmC/lPp/dhs4gmQ4gmMIC0FQBMVM9rMz/5mnj7YkEaxFIRIpJjMAcpxzugC8FJ8cFkKSoxgoTRTSYzApj4CuRpIiBKYiAiKwFUA45MIYgBtBQRqcICxGEA4hGPcMSlEYA3AjKFjByXMxgnAYgxKcGlz/IbFt5EjyXr7ea17onoiduu8dfW+d2m3H/Bj7ugOUd9mVW4XDFg97Me80ELYwl48ZjGdZyRy67HJuD/WeZc8KYR4/1V5lgczjp8ajbNB/Bv3nmx/ijm/RYRFkRA3uk+kU5S2VsriqOtsBKggSspELf5OcYrwlL2FxPX7syMObdD19d5HZ7vLoWeQ9Pju03R7UPjmUxx2MHn+HdY8PmboQtqfxUE9m+/MBglPxt3T6qZe20O/ufOpHaxiHORiMZiswmDEoKB+04VJ36kXf0WHZJSGsS1rwN+TbX1lNtK+CxVjvo4cDGN9MEqO9ObJF9bNPqRKDIvEchp/MCAZBcStv0YF+IfxYm61XHbdDTYGk18uBMsBof7mpUhEV+ZKCtJ0qWLOl1GWYtpNXh+YZ1kS2Eic7rYFh+5kARB9UQ3p9aE/d+d4uqrjd6fmBWU0nM3r+dDL72xfnu5n16qUs+EuxdQlJLIl4lEsTGNvTynCIOTJ0qIhE5slo6ckTT2t+W+LyueEZO5UbkqQC8RcKCRF2vTSzwPXS86ZYFqMdwkLn+CsL4uNiTYmbsF+e5LLdpJ2c79sNGepOw/pMqCQzs/ZxEwBfUugD9UN6lNNe7q5uF8znDnkhqwMhh4YMc1YiJwYL6VdmcApT6mZxrayHPypbaSNGCLHHEVYwhIac47/mY7tRRyFGZLnL+W4rtsB6KoyhVX1sd3EnAqtGkjcICjvyJDN5+ey47lHdAeJuvThG39FAdzgyT/t4uzHYH/yef61M3o1+t7jzWEMR3C7ne31gdPY+9dTKC15/ugfe81fHW2AHcPY+T7bdBM536xXhnr+TkU2cvA6wgw9SQXWnrKDg+26x9ouEXViPDTYKOgb1CDIE0T8v2hG+52vfhKEIlv+d7Lv/E/oB7bsnT+YHM5m9+b/qV343eDoQZduZSwKuGR5SPlEUNwlhltnjRLJUqijrhyQLuOzB8uW9DglryD2OkAm0HvxUBxmlcr3FJ6rC8baH6io2URU3jeGx6N1uN9HpfBWX8GIglOx6S/4iE89yPmtFhMLl6250GKlw+dbXreT2+v7yl9UN9oq35Le4e2wndvJKZdtpgB1hTzz3Pou76wMt5Ry7gtGYyOPG6uZIDrAbjI6L3LJN+/6SA+xKbGTfV7LewgF3z++T18kdZBPEy30lyy0ccFX13D45nZxfkZ0gXu7rzWoLB1xhUqFcPtfLCus9xw09VkWhUhrX+e0Dhvs1Zd5YAMHqhwmJPviSJVoaU+GsKZ594xkJl4UgPkUR0bgUxukO93STe9rNJ5NZo1xASsANv3u8b98KN/JrF8g6GIheL0e4kJGXw9AMEVXdKcXXqoOtggAHtK6XZuy5Als9GInRhBHM3Uth2AxC7J6WWrULF/ghAlauet1hCNZV4HxvFOKYaa9lMA7Rp+xO4CuGFRc4FQZ44NnJYCYBfmTFKqqXXA40edI+vcw2uMsu5QuV4RLC7W3HeNtlAB6ouP1syuZ6eUnm7zntv6rgTJRtjR9h9RnkeqGSKPHS/nLzEthllisMdQdtsIrNuagjXz1mrx62GyVzrX5ppnaUXXQo6rZ23AfNL8Q2q7scm53vbyA/fP7bZG1WHuPCh1EOzzLwcTKYPAUFcvR7VGY4hdaY2e9RKr8o03M8JSEPQYCvJz3mwAUy+8WJhoXRl2dptJPzj3MFgWe2HvSYQ4YA93kW8sVUVObZ/EX26EVVlOXZJIHyH6GhhS2ukoU89Nv+Ztd4br+xDiLnTP8D6bUFEO4255BTSj3yH8igLXjY7dHhtstat5CTeyASbnB/HbaHuPw3LpFTyMkvlnmbD0jJt+fMt5Gxpiow3WY0ECrXU+6h0x6q3QWGQMgBchYFRAqXfoNHc81VxU0jrZlk15kX16yxXdYxd9fWxMtZJAyZAQAIimCASOGKuMGjuMAPzChbK/laYY2NMI+AuNfOIjkEEYDSkQAGiBSuiBvkR3PNGfXcFWLLPwGpqjn3kJj8cj33mc0XeLV3v/rOc01pZ+q279t7Tsw2dju6jTiciwBz+dqFF4gIgQNKfJsEOVJ+5JZxP2KMHekCM+ccpLanNAgF6Vzp7DjVHXE86RrA2/QssF3dM66QADU+a3ZP+zvvQMXM+3inuaPHHazaitnXqa1Oa2cH4H7Z4wDGQxNMbPy3e90epyBMahd28svRWTPYJR8ozp3xqu/DXImAyOcL8Z02jfWe8k7p1ozcFh/thagcicyoQwjb7bJu0mcGGlfWWCUfrLm6bRMoLuDMqXwZVxFQ5APnbsd0YQfiNkxLt2K2hviwgO4W6jiRCuqz05XlOJXjVW17YkgSIOcz48QpG8ReVevoaweA78j7/Mw+jjPMbQDUs7SO04+ZhUBlZlCdCSVEPk+IvvO+/xCdPd/5nmV6d/+f0rXMlNKNp3k8rtqETmzjfumy2842TsO4LiFpU3e9C4ARGtedEbofEXeM+ZkklSxtbAM6SVq6o+cBUW38ehVk/v2FYAb5LvxUn5qo6P/fV1wgnUOly8mBdDgmkFHUOQlGxiWfBIRLbAsvjxfNuDoHFFW4lKlJtNG/9FPtJE2CfyABrMd5uORf6WOkimZAycIUQYCkuebfYBvIAeuTTaMEzeSvn3PGlXzhUShNiDTMz8K5AMh1OswXjL787ZpdFtgMIG4oIu+GNWWZ0fkH1EMQQIdcnWQRpdlOVICfcG8LJPRPPonfAY8rkfWeRECFz+aKZOaCNCktTC33awIyEmFoqVFdS8wL+5UT3Us+QuUHUFNjTy7ZMldMKYzvwThg4wh2OkA3RwKU+Wxj/g3jHmZuImP+DfZLTPJYaSdSPKVng5IstvOLSUW0E6lsw1HtzJMLvQ8HnowkwiSRCBxnIDs5T4TOa/VrGlFYGUBMlmWrZbNsKC6Y/DrcRse35ANUxQ1GM7Tdvigqy3CUH1JOPOSEPEiCNfVSyEkMRJ5HuJvjgBKAoH5dS3Yj0hh+VLWxDelLRVO6YMme+1BJh/2Ax4aq5Jd4gp1P8BLwyD92eg/VtuEqP9UsTDTPwCORQGEp5GQGIs4z3815InLg3pJ6DxJQryH2hFUb0kweoQvWdmwnCdYn6j2q4tN2AjBz+YX2LfhdPwaCAeSv9td2hI+xhrl8IEkcgcVSv/ZW9lB99IZB7wAQG4UVF37q+V/tE5jnDHHinQs7BoPsclIUSFgH5iH63PY+jaOG2V2GUbl7MMjuBr/MkgjJyPLw1BU2zG701jCK6yW72GYTZ21Xk9lgBy0kMZaHppgbk96XvMsZ02BH2+PQF64mJzZG3qOEbD/nLA9PjZp0JKsmUi5aazuNZm706G2Y19xFxNI2LCcGI4gaYNnFeuauGoswDPLAUYcZVe4tM7Tnf/XEJPehKCBiHbD8/K92O88qrLURHruuqW9MlxGztA3LicNwEbKwZ54nBRV6Mg8WTUwuTNXCAUOS0BKghuV8wdY+kAFajtlp5OilbTxxGOaTUbIEUajYk3mQCLaWFX9tGXuBQBPE+R1Me1nlC3b2kY+J7+s0chJWqPHleLX93eZP+4mm0rInkSCdvGcbM2uAJTB3MJs2UGzHH4g55R3UEjOvY59SiTsq8ju9QyNqBYIlyVupL6/LWF/ui+Xn7xUVI8jCExMNaZMjgkt2MDMrVBE929DUnfD0BadKedeE3Ev72KeUdEeta+h3Oo+gJcGmVUTzWRkdIWK5vxwnEFeoWez47/AMHqX7lR0nTVQXOHABtNjPgbkJashUm12S9O6CfUoJn1NCFUYiCbpREAKITfqCCzuQpsRxxAhykleoocePHNetOR104S5waOAB70SoVx3QIkfeKVHJCvmUEq4TmDYBISkVsonoAiHZIE4gr1Cz8LS6RJ9MEaz6AY/23bYsSKOR3L8NQD6lisi+ILrISXbwP6jd0sS4XlN8CZwbqJ9qU//ABVXIN3n+V0u5sAPpDs0DgttQD2K99jAqqt7D00LhLOhivheKL4ETfUJj3mrapzQtfVkSJb9oYvL5+zP/VL+WMjH50GdOCdCz4PmZvodRFSeuXStfAh75yLVrP1VPB+IKNbp8wS+xUHwJHNBc6VFoDNxNlL2xT2kqJiZBbbcc0LsZrPaTUy/zeg2Yn1mHAn2MPVQfLamwQgkqaK4iNBDPufoH4QQCsZVqRpt3SzCz3oPyKZUD9mP+0ZkUED3aiNUMbWMb0OE2SpsTk6D1SO1nC4+U4PWIenL6AWEsL0L+YWcfvGA9GGkJnJBNyJZ4HMcQf36bpMuGYLDtVdxWotnLNilmQYtxDQftzHYiGE9N58pUEOSJIKDV9tzUrtZbOVEylydyhvusLmSjpkvKVEHkiSCg1U7nKERpqMN7gSdRyWdi8i3pdrv5V+9xvSx11Yu7sZ05xlPTZWUqOfJEENBq85gOt2FBCGjYx5lot2sp1jWsvaYuxF128mvcHPkLrH8Xz8eQkjtyptrt2khVFbbZ6dPqvLTDmQ5ydMIVKGS7XbtAz16Lb1xzlJ0uKufCfCz5ATv5XIERMAXnLsfpbGyT9feJCrLdrl3gPpvYWfZIUTkWiH0v9MzButSYAM4AsohBsD0b6WxsZwhlminAqWnsy8SIicl1mDCpdrs2Unwju4tnuZYMuWbhm1uwLj0vsD9HQBGDYId77gAktkxqDIM39ktG/5BocSI2J9vtWska8+KIk8y95ORX94xUZG2X410zO3cvM7CuQdzydPn6RZzA/k4X493C1e0q/s9Tk798BvhFrePjEi21yHa7trLbY+EkczJL48lP70n7k6oy60GDWYTGP1DEIzKZJWZep3oQJ7oBpyOYYk9WFMqUVCJAapLEcuGnhDNrX+ho10i47t24T3z7Otlu12IyPpj9QQZGj80OT84e6+PcoYBgJG/3QuY10++Lff4bHqLjfMHBeuF0mZ81oOVJp0skYkPI6qqKI0xQYUglEUxJT9Z2LVFCTQBCL4j2uzCFhWuFOLLOxj7h+7fJdrs2k31k+bjwZ2GXuyWEJ6MGB+7BHpqdGYq9GdEiyF0omNfdcXJc2cZ6FIviy3iR0B6+Yab5mxFCqbhnJrAVOB2BCMRPgZkkIoIp7ckKoZVIAKkObdcmhOtlQmyDYiWR7XZdBgQXIjwdO2oJT87uwYE5MUHBUMXFEwWYU0/NC5hTRvdL17YA2jA+jRAIjG0GClyR4H5EmDkiQmkaT1ZSiZw25dVECNcJ0L0wRcRKwu123QYJhIT0rQqaIIvPEG5mq449+fTKPGi7dlFmOE9HuJg1JLOqJ4tyrqR5sZl+O07ayfidluxvOW5HhEQyBQHgjeX+bQKQlJYIGFo7jScrqSQThHBj2xE6bMIJwVV3Ok/0w75ywWDrsD16Pd9X7gGCguE81Z1WFtSqsN7TBNqR0LpkBIEo4kMZy3EKaEUdIyzDyolEHrEgUZJ1AnfLV0Dei3O9hLxKgeesJNYum992GR6C4c+mRnXGfGuSwLchEP4eK+qchOuR+i7+7Qk2rYbFZ5wjS+HJKlGSifmZtoVcYb64ZTNoT85Qd4oRBXXVdqNTE0pW7YIeDMHw47SCaPoQTpJHnqwhhocIO62srqp4NyAE7CaJYEp6svbrmlaCu4hKAyHcKhpmbpq9TJdsnwgYC0xInu+gQOWOjXIPQDBMJzxK8l+7112hRCEFuC+EC7oKSbbWe8CnUlSUUtDCW6KECHAqhxBuFSUDa5rj+3O2erX4In1E5SQwBpPrDp/Kq06de4gvIW8y4Rn5YJ/fdoe5ipCX82W4q5gGK9yTe2m9JDe3I7RyddM0cMWVw69UNut3Y3tAoCQwOd+/DH6CB8EMM0lblsTHAesWxr71k5D2h+PKdHVbWF/zw25cWRSfYrXeummuSrCibFa/5/b6bicZm0FdBFmCA+arrFHJg0oTDI9tGikpH3GeCBsXN2WRtNPBGueeWcmQLhlk/X/++V/OdzI08M2y28v+LXhFuTu4cwea2F/vsd1QZjAzCknY2gUPHYV+j+2WyBe+xJ+/8W7QD5//3T7/7wj5Yr1zRGHGYcriLgVaEnoH8zPpAfZa//9Pfrs9jkq4oilHzw3ZAcqKcQFyAIFsmcA+XUQnF7bjIkK+hI0TwdyTs/DGBibnQV/5A8cK5ZdI+a7Nuz0LbAeanBstS7ji0Ck7mCMaO+HAJ1lAcHDVqlHZSdiOiwj5EoVgSfw5JDQ5HvSVP6AEjJBMErcjzF2W4cM1VKFCNNIlRvsBG/hmrwe8vb4/aZ92B9egFVfvMgUDfq7z1C6xwiNViKaEuzWiyfGgrfyhPsUKXt4H3p2/4Ma9JLiRLnyA3OF2lonMZv/eaoOe/V7csh6haH9OPVf3FdGWwNpewhUKkB9S3oIa95IQjXRxO91syFccv9IwoxpC62HNsqeFu/1UdQWKN95PVQzh0hw6bQlEe3QdoMZbJEQj3fmZhamspLPFs0R1yPFF3ZhhaQcWQ/QOJKI3bTRL6+a8UreT6jz/qzWwlAKikS7snLmdidT2Ctf7Rbxi9e/NWoW78x2kH3UZs8LO00YzP9M2nonJtjETZDAHp4G6JmsIz3zP7fXdGpxOU/gOnjL+2A9aB3n5sHAEJR4eSfyvcVrApJWFnjQh6eDWggobS8//al2rlyTjjW3LkrwWQA3hIVxDOIB47kATK8D+b392HAryVRDL63UqV22J/3UK0sqC0gjpE5N9uK+4pWjcjQgDQzTSTY4C3bH66v0uUtQQHjo1ZDruCLt2VegLoXxUUsZ12kGAuHPI7gLwViXxv06DRJYMQnqy/v++AW6jrqneY2Gf0sIUEQaGaKQLSjAgTaav98hgZA3hAGKJq0NOGo2rqjC4Vugrl7rBPes67UCgjbe6cD/hqk20F08NlpUKQjrobwCn9cxkyUMa5peXvoZwy1hcK7P60NTddYdf2/afywpjofhYQrNGFYz+NPLorvwf7B6BIFy1tYKlAwltjJbVQK9pfqqZaggvGYnrg9puOhgNnW0XWn2+9wqjDYULusmI8OAuqmCYbGwTE5jQdE+gTUyeXZjSBUrObQv8thepGsInrKdF43DFamjEoeSUBW5OECSkC7rJCHtwe/+EdjClXLVTgco2gGyypOl1KenKr49UOq0fg8WzpOOSYbD4GuDSO4DWU1q6oJuMCI/uUNiNeBa1sU0M6lgCerZEklIWfKqLpIM0TVVfuKJfj+w1hGWz+vfGjaLKgY49pAfxhpqEB3cYTHBSYeaHcL1HokQCerZEkVYWfNJESF9ONExMrsN6Dwf2c3MX5bj0PQeamDzRgXYmftYFEB7dlf9D5KvQ6KhNNFDicUa/T+pqCEfPDRm1umKwfJ9vNx5+nKIoQDVhGFAvrRuFGe2mwD+pLoUb1g/a0KlRsobQgN0grw4m4jpaKASMMm7qAbYLBCYm28q48FPE+AfUzt9gu3YGimsIG0FhRr2HDPs4l1/QLcgwg5dPY7e5A3U1hA82JTMYlcAaQVVoqheXemODpKI6GafEwGblBksMtmPu4xEE4oEDIjk60FFDWJZgd9Hq6R/u9iXEPRbMLHGbaJOIRxAIH0QTcjSguoaQsX3aFxZFcXqsvNE5V8XpCmQ6TLdJBLWVuL6RkmM24nzcYHQJ/6WdAEd2++LEdebaWXEcg5gBRO7XZJtEUOOOt6PkmM+zirVd1jK0MXaZBTHQKarD0GQQcowH73ZEfC/Xi07j0fClT48Soq4lE0qODcBC8qP/kHnxGijXv/Oak0wMRih4QQWUdhqAHNsoiqPlGFxVPuCmjau58AV4qhJtEmUQcuyhKk5Qqx9w0hfPBTBiu3WiTaIMQo5NvLP4uGT+sWvjjxtDUc25xwIUjWYP18ZHHlg7jaqvWwBUeLxX7jInYbf5L3Ooh3yHWeUgbDfvhfihucqOfGNROQf7LZ/xiXFojTuHXc4ec4CVPswa85FFJ5lz2omcYkeYF+NL7WoPNBlAdARXTvkvfuR3od+GzjVNyc05M37uHoXPxmHR7xJ5vGm4aVoGvFLZLortz9x1tUXAvxxtmJurl+raDrYcZYcgq4cNsibYOPlgw2VzdrauLcFdBjDMwoyh7wHFlcNJh4yW8PdcxpW63hO9aZwNL8aPNyU3V0TSs4b61GFn2S1pFDfmS1tG2f6c69Bi9qp4N4B4V13L8EhzCXzQK6nDP9gabZhLUXgPYVwY9ErpqDyDU6ug8D5e1wGJmlEtQt2o53CPYGgTFN43k2DL4Uy7wORDv74g3DZSx/llJSWV8V/QlNysivQW2LirifPLb6ScThavNgwL70GOCGaKAv3+h9RxfpVP9JtItgYQQeF9sa6DHNwBCvT7P9PG+RUn+k0h50Dh/ReM13W4U203PtFvznl+XZzoN3kcaEIV3oOfFcg5z69LE/2mjvGzksJ70C9w8s7zK030mzYuXS1F4X2rpizgN+Hknef3WZroN2HcRRTegx+KOYHMeX5VAHQTJgA9hfcArwdsVCLfPL8uTPSbKg40OQwL7xlGIxzg7UQkcs7zK070myJmYeFdDETYFSxuuckW87AdYaqqcM85z+9zyvrFL4Ea8NGtunYHp9hV9vdytJ05Kkj5VJRgpGvc1MkzDXMjCu8u4aKzuGgsAeGHNHTuCi68OwcnLQcrKQrv3zNU1w7CST/y4GX2bFly86AYzX9LgH95CBfewwC/7zcWF97L5upsXYcCnt+v+NC5Rrw7Pl5rs6GtrVEuj29tBRLPWnOSOYXjROFdH+fKplxtyoa3QgivGTdHVRVFMsYQUXg/Weu1oYZL2L1jAHHCYYzYTiXefaMYWEgz+i9r7XZLGuGw2E6GD3NV8ciIq+z66T1UW01VnS+KB5qSE2DhXfsMzgOIuapwlyUvexHbH71e7gRgLhtTfuGbpZjbyvwMMz//q9XS2cd7WkF4OEx/4avxkBfbucNc5eY48Ba8O/FpyZq6vH70WQJTZqu1ncDBwE1MLkypZH4mK511eDzio/UcwX2WCfPf7bJWbH/tgaIoMjD+k6NqbZX5zKh2O/QvlUMM6Fv8rZvEcpsYMen8TD83cKvZQyXerS2OMY/9w28ULeizsMk22iEtLKPvWWqTY6WsUj3lGzwNjHuamdsgA/b8r0aaiAwZxcTPbqOxiMMBUucErjUpRhfeH73B6JJ5bPaEeDeXicPM33ZMqa3e6Owx7cY8qoOJSeK+QYyVMuXoKUVyWl4HSsBAfwhN8M7W4TYCfALvR0hJLvAP15xrnybB4ML7OyeF97Hy006Dd3OZuW2Ply2zrCjIcpFDqEw3ekqw+TaW0yZVdlCRh6CzTh7PhZ9q5wFONgnGFldudG4nwBgovBfnxVMEFVg2shS9LDPzwhQxhMpMo6fsMENoTQtTEpbbucM1R/OFFGa++0JceIfv1GDXyIN0QeanFqaIIVSmHz3lEvPCfmU0oTTVE5P0avBJ3ix84YiJVjJ/34jZ9kiS/DBGFt6v7QRAhXfwhFkVVo1MC6JrEwyRZIiJKqH+kxHUhJa/sY0AtR45s0lVGGnXuFwzfxxOBOYV3q+DwvuzTqDiSl5A5qOIIRZL0SVAGiiwE7pk+T1MZx0dT358kZl1fxDT6wUpDPvu0+9cjvFY+c6o8J43gLGqg4/X6xRDqNQlxPbcB4mL0gTL7sQEfJIf9YDm1gBCTK9cJDGr8P6sMim8H8GFd9ehs52IKIrjIVRmGD3lEvPG9rJIakS9B7xVCT01AnxSU+1EvJ4KU5nz7uHZMeadAETh3X04qR2h11OdLyDF+cqQwvsjsPD+MCq8l0lQqw1/nR4ZWQTj4D99vjKm8F4ShfdSCSd2O2/sH+U0tbjIzA8XlQGF96TOb6zGhfeSCdc1UO/xrJF3Pmr63yID+7Gi0pc5IwrvzOWR66i4EqNgfgl4ZGY+YjzfWAp2AjyqC5E5Iwvvj5DFlfIpMcvCyodN37gqfgwcXpLaNb2OLn5hUTwACu+zD6PvjlScP1omjH1a8jKdR5M0xO+sbV7M8uL1IzWPEYV3H6HTViAE1pn7zm7FWplmnbHpfSfACU1UxTvXzHTh3UMACUyBkFzC0FXlN1pRL/5pzLPaXmslv8TT4w+P/RguvEcPKt12/uhOgPFCZR8FOr+4rh8uKk2KS2b+lPooLLznI9j9FDmcooYgH30/1W5K7osKPVTJBiBQSN0Rwkkh8zMP/ZSQhoWniKrQbycUjxlf5/3vmrb5NS7LcgwWenIT4H5KOZzWy0ma7PAGQRpf1EQM4ZeKG9zitrbrxCSNo2t6qIoT7zxik31h5sSQ6f53+kShq+HvF64ldn0knexco086nMKm4UsUaXxRBdgvlWBiEqjb2CZI4+iaIEq2y7Lmtfx8YZlVKaMNExBINTC5XKAWZL6oGNovtUY3KuhoQaxOIzwlWNp/7PVshMrnCFtl/7DwxMHQ2ktyUviiAqHIL1UCOAaKFMLTxLMs6qMjO/ooW2Yn1NMG9JkAAlJNsGF/JMIvVdkE7pYkRizqwyk7ZPasEWvsmrnknGUIBaSormmgtCUEoNVksaMtU0lrTLNaDhCeW9gUr0sVa4U19ixjySmlLEJkGeTauEZBOpyih1Ab2xOTz//qiUluy31RYQrFGhB9+W59YlNSeA5hUXwTdVhzVTCXnNwDlwSRZJDr45oF5XAKihkP3b8NXFd/NEmjKXxRgYcq6ZeaCCHqPUgh8H5HC/dhbDkL5pKTX92298QXbRp6RxBw2CBATgQuAQDhmVKF8phbLw46QDaxHWFAYvoEnbNENsmVOEuCnIhcBnj8reQUJBEMY0nuaKoKwINaEWXBBzMrtqczO0779s3OjAzylYvUxtVLQiKzXIxwUiKS+Z2dV13XMY/w6QNAww5z5eVcBYDj5EBQd4X70iCEL+MaQB9v63QoKMK8jajCve56BNi3gDSu+1cqwIHryZfbkYTRdlEGacNAck+Am1bdPRcu71tSj/0TsD/orJj1ZJNktLciOR9t59IY8YKOReUX7Ns4GYbeS7PxaQAsb08S4xrdbuog0BcNENcBH9fPlyb8Yt3t1iFb+41OOwOoiWzGQ8D14cx9GqgkRyf4qIp8B9gSEy5a4PnAUHhE5MoAVGYZixgyQQesliUAwFe+YH4+GDfactc1P2ME0u71iQ7385TrpQfGdlFoieF2mc+yCBVXo0Hv1xuByixjIUMm6IDdCkXc4KjXjj17KxxIBntQKpW8fN1Ex+2mVFyj+Dyjvc5vj+0uMpMfANYD87SSmcjW6UdllrGYIXu2a8vrPdwXO06i+CiHUYzSbK7o66xdGvcgHas7qTyz65pQhIP+Ejz0U5STN/fJQ4BRgAVA/cIUrR4qyVVm8etFwEMhN/NkK7XbMnkbtir+kBlozTIyrBewuF6cD4VLAEiUk4ii0vWmDQA2oyXaxNKe2UAboQhLxEBH7DRBgSVRgFOoz+HJmX+9CMzAHg7ZtWoXGSgDtjMBE7KMb952+tTJ3GCcz4kKOUkAQERgEpNKN60f5FlB+kxgz2wMoejPIokYoWZpYSpFUGA6CnAa9dt5yzM0vSA71lAu1lRPF/1ZxvO9r/onT1aHIqrNbM4SACOaJqVI1RlBm/CpozyzaQhFSXtzmRqYbimXblJXh1kGdVz5C/3vhn+ilhhvkSEvKEFWFkoUkOjIMqaAPIrjIj9V0wAysGc2AaFIyv3bcL8UQYHJKMCp1OcXxTK3jxvlYi8iLygL9bRJqoBCS5ZRDrmkH5hasGe20onsEBJFYiH1JJcZ6k8o8uT//BgOJILxF7WggpmBtkqugKgH1JJllEOHIdlIfa6xnDSrIbLVwPVOBooCnLurC2b+Er50BDSOzvd+nuoOGkfne7uc7xYQ+wGNQHGWMTXUYkw61HMn0jNbuhuC3K1Du3TjCY4CnLe7FcxsDYWacPI8fZz4Ny4wO8KFACs2TsQANGcZpfZck3SIeg/SM1ta7yEH9AibKigwEQWYgFaft0jnF1wAEquSCvdg2i446F+tM8soM7/dR1+RCZx2hNTzJ0HAhUqBe7hnjzPXv5vOLKOI62hlFHlUGU/7OiSdWcYMrpfyCCXZtqeN3iyj3OqDAxSHUJFte+JozTIKckxlzr3ECb9to7KMkRjlWcYIjuoso6fQAVUU2ui0lenb2A6RfgE9hfkZnSSCD34HOKXekWP6mLzLzBtPkwQ5TeXRSo6+DHuqysaVDMV7XRwW756kjsmbJuhvypi/SAT8w56qxKFQ4n04TteaLVeFXAT7oRKep6TDxDJFupi/shC+yFOVOhQsPq84smiNNW6J05WPq6EAwvM0ASewDgXpmLowhaGD/kK5zNShYPE5hWX2qFPiOOYd6WPyLsnBsiQerWTQX2LcyCRLucVcVXyaXSyePp8ZqywdDhontUGTuq6zTICRHq2UCAHyVM2hSdpfnGV2vso85hO7aNLdJN0zjnYToNIVTFTSoL+0LCCUACtGnqoUWHwuMVfZZpkfNu2h05pNJoZHpA27TBd9ApBeiEI3fiRFBf1NGfO3hiLEBG/P0kPJhd0caNV524bNoYtvO2aZ6WknsszP/68IwvOU0sQL+5QWpnDQ37Qxf7EIYYSnKgbXe3gwc7bdI/VwSxrZRjuUDQ2Gsl6G/Mhdj3eYJl6pbA9ayOpJkxBFDtg5fozBJBtquBmqdZpZkw4cD03kY7PkclMrvup34VMGMGzOySw48Cszjc0yodwMDMpFz7KS2bA5mIUEJfuYGfb57uDEzB7mZ7KV04n+jzXSabufX+BjXDSIVfy8NjeYn1FBrtv+nDdrt9jmJTdlXUCGKpQZLW+v705hvFz1AaCLakYP01QeqgAkhgrMggKdUtrpQSR7CZcaPusUznJzMvch/EmzeZim81AVLyxmYnJdJOPnf3WKQKeiZS0h1UcYZW5cMhtvmFdzH8JFNZuHqdRDldBMiFkCTQ37dYpApwKi4ZWHcKlhdsrsLHMy84HVoHSRzcNU7qGKIcW0hQhKCNIOAH6snsIos5g5Zc686g1k8zBN46EKky0lZnldLBNCZCCp6cn5uZi5ZA5mvkA2D9N0HqrUBChY+L8zbVIIDSE1NTn/+WqZzNwxL1fFx75AFg9TQRoPVfRNhJCzC1NASDqQRu9glpsyoSmPO4LjZZNQNvwvPYHUHqZotzQeqkI0EgO243VaCEm/hlJ9ZPIvt46XPLq1daCJIxjZ2hrl8vjWv/SFSTYP0/l0Hqq1dDTJbVoI1i4ZRLKPfNG5hq/WLrGr3JzL2S+y/CQP+s+g/xwcwmEXWdzDTRbzuKt0FafiCPlqgzexyMh+gSPYZGSLbuEMQ2uC+G8V9ft0yWv6fdosmc8knA1iTjYsbDSX+gU0ME7XeB3KM07mntNv7GbJcB7Os1Hf6Vd8qMHzYP74kvfEndgswTycZ6P+E5doqAmM+S/gmGCrlzyAcY4LdtgDmGVujtljJZfHvMZuMJrZ7sP+Nm/gr162x67wlcteY8eYL1tt7+MP2DP0QsCwxxjA5sP2CawZmiei5y2GsPewvQJbhm5LMO0pRmDrYQ/6z6D/DBpPzpnlRAQqAgqINioOIoWQ2EpOKWXMCQg0BCYg2qg4iBRC4io5uYdiQU5AcK8jwJsWbVQcRAohMZWcfDlOnpIPthGHcfUUSE6H7x2sAITBiCrfA01/ddtAw7POSmC2yskQi0De++7Lw70Iiw/W03VmfJfsYGidi6SycRJkDSRayIKAArCgEEB4dMWHM6Vy37cNaDm+OGxUak98f0Y3HAgsrEexrJNrIWVBRMHwwO0swkIfaIXCKdjS6dAsRdBw/DLXMK4gL+2TXe+JWLene96aFXVn0KIxeWkh3mBmBRXmHcAssgKfgFBzAwsi/8qXYuFaGgeV1gcqB966C8CljauX2OXhZPShZqyBhxFduVyNSTCwoPqvK3XCQdjggMi+vGRtVN0VzuP1JIHsdCgwLpAUW7H5u+xGZGBgN0HU3wgRJPWkH1rhAEE2BogUTxAj4y1c3QGxT42tzBXLHwyM/goaqLIJ544STGr+y13l5LQRUIw4GdO+JYivu1kh5N44qP7rF8+TytVDwTDH7wSOGHEy4RBtSV5i4oGaFdTp8GaIo1/mtrWh4xqhBCIG9XKU5RTiQE4/PE4oX6xxj0hGXwYjU1LlLpFpyWIsZFPwq6RuOGqwXtQwrjw6MukiWIEsYtwj0rYjTDlJ6aR4+Z79hYAThydc18bFR75VLEfaduZkcFI6KR6Wv2xcwInDE67nqHlTsdYPSaDgM2MlKjYBCnEi8YTr948B59XacQLQY3vZaGM00yDHpk1tlAbk2HSpbT306zHoP4P+M9gS3jOOxRDHe8Y5Guz4jGki5ywA5FuYvGec1AAdzuQcIOVbl76NvcfKBC2djTAAT/m2pVvTPBDwmO/YE3hQTU5sw0pYywpWabtFCVw7/cfEXDfy1sUhncwRGMC8tQdZb1oT7ESgZ92YrHD0woPpngfZtEaU1o2bpcIx3PMqAsL0LoTrT/5g86eduW2QbjDY0RLsNcJjEAoXBXFDVM+ahMFmhtBMXlqjaN+O29qOq2cRCN9gBOPqvm+Pkywv7Rle8ylC4Z/Qv5obn6rTAa+FBL3hyzfN/pjv2GOxfPKxXh7umcq+NTgPCHrDl7+F/cdujiedDA+nQHTZfLF/W3Pr0C3bxLzLGq95/23MX6oRYXJ8sYLyDhqsYXmFBv/Pa/Z4H50gW9WEQJ//w3HaU451F1iGe8GBx7ggmUjyDzv74EVAgPhe2lODBG9xUSUw4nFhApSERPo8kQDBU6q3dTLdS1N7ksWZhOo8whjOdKradXGmuTbPPMZi+LEfbA/zxJTLjNvAArtymTGbHnqgq1xmvCYPowq/nnKZ8RtZBx76o02c5NYlXS5qPU3ni/p9ChyRcAVNFLL9EHV/CBDxRvCc5JalniYXtZ6mniF71O8zCxyRcAVNFLL1yDrwSPqjTZkj3a6U03UfUbsG7ctSYznxf59IzQAi4a5JIiDfdO3bcdt3f/3vZvMFwiTCprRx2wkLfCQBKaTIyRfD0O9bOvOk3IlBAIfknCcfYM0jhIpz8zRfHkkSf/P8umNPtKnlhCLhK+9CERRcLCPyKMi8lEIt3RjtW3whWaPQkM6Vzo4v+AIfEtKtS8dJuIoS74QMAYWMTnDOuRUwwig3jxFxpAwkikR8GS2hWBFkXKGMGWFcItZukLIlvDjM8LLX9rXJT+9Z76eDFe7SAh8SUoABBsCVpDtlZPAE/+idMTFAgVB18zgvx5ScM1EEzJQUNEexIki4vIxYA0KxOIfC07ODL0Nl1F3jzl+BeVxdfEpVQgovBKcJ5CW4GfBzHn2N3peGeBpnHJTuODkjrLvnrVlBMD23ngVwfd8+b4EGSjO/w/VmZ1Y4SnzZXTNjb8FbGiQScKUyAulc0U8CBBVqSqil4Z4NSCXFwB9FktKJSEE2EQQv/113ZAasBRmsH+A18XvOAYACYd3NlZcW0Mpx5e9DDVRBc0UA/LWnuVjR9i0Ww/etAFqQSuuhAAIuLyP9lIe6CuKgvKZYrcAvC5jfp7+RNK7X0PjiMFvNxNwkMrzEcmJ+WX1XKRwC4dMOOq2F+GL4r1QIsW+RhrUnC1cAJqPdt+LLsnP/LGhgHwlXqI0S3rQaTJiDCkysVm7Cga8Ki9XNgi4B6ZaWrGeC17gHiyLBk/ju0NCUxwk5mATCN+UXw5f6KKwsAIkGzC4rzhZAAAoWNQX9kLuJEi6vDV41QQKoyDSuN+jydDpAD33+9OT+q66JkIgUdpPBuGEm7qD3TgaxFM45QAJhlEqz8WknqlR0Vzmlk0RmBTSZRP9NA8bPJdXG0lBVyaACE6uVG3PAFTuYNzZXqhSkMFutHdxZy+AO3tX+lffpXaU5h3iQ4gonHOLlOxT45YJ+wDCM32l7jzm4O68lC1aLCzzBYJ998G+PkIQUeLvR9ZRA8AEg9l+J4BwQhRGqEszHB2OSrK1ULh+n+MvcoCVcVhuAtaHj6vJlgYnXUoId7cwTj+cAoKSLQupNhxVaMuhxsx/5GK80Zi7ObUYg/IsuFt8KQcCoJCMzjUp2OjRQpgW+IUsCrlQbvliD5sZBBSZeSwl2+CFlZagzK+mikKYDDTaKwL+wHcxsZr0MTwKHL8K4x7gqB2m0olrmSlDUuEd/rXEPMsyocQ/02oxg5XEPziTUUs8WP1UeLwOAki4OaTqc2hGmDADi6LIOE3RuJmSKcEREn0XEVZA4MfP+VCKNJ7YESaoWKR0nTjpb1OnHIHMXIssbc6owyb0sHJkiHBHRZxFxFSSJGIxLJdJ4YkuQpWqR0nHirLNFHe4FF0LeuFmDSdJiwBFOiQiilEFDYiiMSyXSeCJLIJVYhJWIVbZIgmbxMPvjTD/VCMZ0D2Z/4jZanuN4IKHZr8edXYmlcYlsIprGrQt3PMYCnT1zOO8z9sR1DHH8x36zBcTVcRG0Gla3HHZrOdR7tB7qxQf9507WKAs5xZNFwqRQAaRHX7K20F80WY6ESaECSI+85KQuBBtLlnIkjEwFlhhLyRzQq3Hc2gEdtnuMB1OQGI+yeCUdByJRAXvBx2qk/V1mOacoynFy3NGZhA4btTHMc0vQXumLG9MpbngYRiZCElP0h5yUnCIpiW+vxRw2amMIQH7PVeUGFFAIFW/rcor+8l3JqRdNqbt+3+I+Kur28OI2Lr/4Mg4Fj6FCqoy8969hEXuOiF53me3jDkCrtJl7VJ2wjbPU7eGft70Dz8i3PpUMBS8N6S5GXC/UECHdgby6K+BV7OXN3GPparaTVt316vbwgME2VyRCwUOkId2hRBTSnYjj/h24gbSZeyzdDY0IbicJDw+AyW14WBIKXhayF6kGisQyKsvUHTVrCcoQT1dj15wrRXh4dJuiQ8FTvEMKcEj37zIMICpDXF2dkihpePi/SE1wnNNMExTSnXjWxQjFzdyjKDlnUdLw8P+FWk2HgpeEdKegQ7p7StINHBxZNOVe1rAkPDyY4ODsOBQ8ggzpTqmGId2BItasSWW4O8oiDw9P13sQoeCl9R406HEujuOOB2kz9xjajjBlgWCYDg+fql6ckELJTRMbHmqSyOYQw70o2s48SwS9FOHhJe2leoQUSm7K2PA9iaIsQPTiqB+STCAPDy9pT0tKoeSmjA0vUyRBRFk/VZilGu61EPy2ezBD5XZaEP0oH+zUz74F9wtfisNi+W3bL7DhfgEYwOJLfcb+tcDye6NfYIN5UBzHBOvvjf5ATDA/4ECTMi5wSxo9gPq2Yx7NAXuQ+cyoH9rV2ToM7QzzxbqVYIsNcznUSuAsJ3xPC4Ghhpl5lzMtBE4BmkutA85w2TTMZ1sJWzfHR5mblsHG38PI/mUL4eOr3Jyr63rQfwb9Z9B/Bv1nMHO+h5FdahUw23Bz9cFWQj/K9QAinJ+tWwachHFYFlsNcbrOtBL2/AVifqqVwO31vQRbtxJsFG3dSti4WWwtUDdn6hYD57bqp6BhP8mhYFdOBQL/ksPBHgwEtpi/rwoCK3k1HLg18SsVCQJbDQterXwHgAUHr3b/9n4LEF71/u35Nug/g/4z6D93siDGKDvIpi2eE1NKUXaQTVssJyZ3+iHIDrJpi+PE5Pt5LXOeqIPymqFcpD0bhKY4D7TkpGAB3S4JiAKucAfxmRVoXSQACIEIMSAKuCp0EJtJDjcVwPfjAAZEAVeFDmJDnCTixbJDZuveHHbuh9suGgfLlLLn5bGzzXIo12uuHncqYYqL7RmJmCCBmi+PwUFPTXC0Zc9gcGs5O/lqfXe+suPldQReXhnZDkdmtRow5LOxEY7J20hD7PIbjcgnfUfBu9oeOi/WDNm4WeEjxQ5fDj3f1lkFos8n9v2U17bQU5McupVbUfLkb4NcTmJhlXyz8fZkJMXezfDs0+WkAlUfrNxRGuV3cHUL03eclrfZ+SNT3w2XQ3mysCFpQOkzE2J6aoyb8DuWk++KrhxPqmW7nCa663i+Lq4AK+RyaogTMbxx4n6T0d8hQmGot6Hf17RepgK18ap90UY5Y+ePd5bJJJ9kE74PNdrWv+l9BShCqHXv8c7f69gw5/ZErm1BlO2TL2AfMHFYtnhSgQGfrXPDHF7U2MvCm9ZZJPnj1+E44OgI++kImzkaUPdZTn/zFBvl88ypnXLeB3giSpSTRCpwXECau/NEuhot/Q9SnPwXxdmBynmB6EGJNMUKnCQgjd15hN0VjdZV46eqjFoQ6ANfhQXmgyDkAZky4gkiN6XRtN4yHsIpI5rQXbn3tn1+wtprs1e4JK1/D+OUEUmIK/fu+1tZn11NsO31z65WuuyJ5yE5RdGEZ2qkspAzrM9eXu/JOYomCM1abNVKVjXy5LcltqBjzCTWUt2rxZ4ZY1A/ZhKL0XLJiC7o0WwzOi4ZsRsNp2jQfwb95+AsyH4QWgN0yX5yurEL0AWN8+iukP2orXTjFl3YZYz9hGaF8INIrcB4Lysr9/4xELEiZtFdQV3GWE9oVgg/yKCAnxGHyHq9Fz/o56Fuvm0nNCtUeBBaEQfyK/7SvTGIzNf75JWseR74sxz98nijn98p0gyZte63Jwu0A3GCnxGFUHG9/yDx636G/aiP5n970tLlKRa+qFEgQkMEIvv1/umrfzPh94o4cpaDhtsTnhXEdfB86Ds9QkcEIvv1/umgNPA8CfrGTiWBAboygLNCEh3w6qMecQ+NQGS+3sPvEB+kIvJjd1cT6NeVHpwVsINKETFT4w+pr/dZkV7YAVxNoF9XGvSfT9WArT8o/pD6eq9kgRhmRoxqAv26UlBt+Tnx8rNlBebpX7EelSYk8k3sI9NuPOTuhfc+Gw8/KLkDIeIJRx8yX+/pL7Icff8bGHnHKhORgm+LN7wgZESf3bJf78l6QPtXS5bDKwQQuELFT4VnBsqOOfUe2a/3iSLcTsT63SRL7m8AArtk/NQu0dw86pD9ek+2I7S83kOlRgQQSMZPxbI+LO6Q/XrvlHbmgVkj0+FHHjJf7x3kh6SuzEUqlF3vXUFgBS9Wkep67ygq/GijFWmu95GeLJoH/ecpRxHFxlXjXKL2uJs0XDIQ0XpcZpWOS0aM4ZCeUV020XpcZk+SibVZnB6334d1V14pv3+bPm7Z2Lnjuoxj40KOnON+dergox/KKSNyYP/g1B/GKSPMcak9hFNGoOMzdhA6OkpRcQYtOX7DDsLsIrthZyiU6ep5ImTYQRj9SNewM9QNY7ormmoMzD6IDMe0/7Z9uXJ7xWs27AytdIMQIyuWr77sX0yOJ31SuadSbrhliYoTdF/J6u23ZTlDv+0e5SbOdhjzfazermThQ1hTWlHEM1kD9ySkPYifp6vdYRDDOiwLe2avg2cq5Q/co9RuXigb92g5QcGMcseFP5CZ++5Val+umi+/V6ndl5lf/+VK7XBIo9qxTQVKj0IDSo8pM+ovGCGNYsdnJag8Ch2oPKbsqL5ghDYqf2BqUHgUWlB4TApQfMGI4qg9QQcV0qUjluqa/ELBCQ98uit0RGvBgRJyCxUnPNAhZNAjHhAeJMEZKLiRqEbJCQ9zsIo/xmwma+OAq75HOLAaIEEd2TN9GlF0B1WMmhMe9vymr4cCOUbnORn3+iPET1wZCjJ9+lB1B1WLohMe+qQEpQNxHG5K4U2ptsQIFW1ErEEVEuuAhz8/HUdRd5mZLRAfrPVUVyI1+vQQsQZVSKwDHvpgBxQoLGizhTMYj6S6EqvRp4WYNag6Yh3wCMBwOVGgQEX8697gubbEChUtxKxB1RHrgEcDbk/o7xHyubWsqy3RGn06iFmDqiPSAV+JBgv+01EMnfWvd+Y6X0Rr9OkgZg2qbiHWAY8Hm/Bt4rpGW/xDrS2xGn1aiFiDKiTWAQ9/tnUWSdwBhYsyqzGxGn1aiFiDKiTSAQ98RIGvHNgC33B69omXK3UlVqNPyyYxa1B1xDrgIU7aqorNoIxRyO3JWF9iNfq0fFG8GlQlsQ54NGhHKBhASKCpK3EafZqIWYOqI9YBjwbtzAUDCNCutkRq9On5oog1qEIiHfCI4IckEACp9bzaSI0+PcSsQdUR54BHID/VWI0+Le3MI9aghvmpRiDiNPr07RapIunGRiI1+gzvxyAeovTnfScIzO4n0zhiWb+xdvejrP4ERct+xVWacf0oR7O4E2bHYYk3/ew3lVq7YnNcIuWWmT+lNlH/gYCmYfVWZuEeo+PW/YgOfl6CMScooPlzlYZQpJnimt4cg/qD2Mun913pKvtJ//IvV273ZTlB/+yKhuiV9yVEurjX6Q/K+hOUvOLiuAg+rGv9CVrpxsZxc3yY9Seo23oYZ9Cg/xwkge9bgOD/FhyEgAUFwWDBwKWA4EwgUB8fDQU7dFfdcrBB/7mzJeOj9tgmxS+w51h/EnGTKjf8mGV77CLFqH1tmG5S5R6Uxj332mD3STlSWGGPGQzJ7/1GhHDEAvecVDx+qi0wKekGhHDEAv/WlDx+aoy3FOSdD2GJ4e797uB+x8NTEEZeckG/9IJUl8vJEB6vKIpL8kT90jxyXWbh8sJ0QEA2KxpyHo+cIOTHKeomuGQbdqAMuBS6TMI1Y3FwAQHZLNeP7OVxAFuBkPLjE3XBOBRJkrD4RK1YIDReIDguvxUgBebZBEkXKqXS68Vw2V0e/fVy5OrosopmQ93l+5G8gmQUyFS2pzOzXhgEAd+NAUgk2gqdppLtFJDCFdIoxWSh8WRhqCGRaSRNoX4yv+2oJMAdaiLWbHOXEJtyIIAloS16Fe0HTF5a7153DR8lh8YKAbJEoq7QKSqlXWKYGlc4k5Ri49yFYltnIMh2GklTdFWvBlhEQV6h2UYGHGaDCpIfs3XYUyHoz3fbfEjrV7ctHANYJwIkgeRedOgEKiGeM0yNqxe8JBGfsJjdzN4nxmUeD2br22uZkcy5+Gm2CZmgsQwMA1ntbiIfd5HZCqF13eHXtt30MJvVrwmwO7aTWUOGD8rAj+RoaR+qh0Ndq5CvIEvDsj0XU333l86sIIMqjQB83z5vzXrVofOBU/UYb//KOzauDrSNc+rhXuDijmPMJzIMXKxo3zmWK7rZOg+kGGwkYitlzjQUPH1HUOJPF3WnVd0Bl3ZRHcOKwTiTrTpQ/I29gh/JwcsQ/M73tQisPUINc1V31vtgvXs5V153RdBcAUSCTyAaGNq+bUS152JyoKVyCNTDQYvhre9oYTM7V7EL/zwElrPLtR7zJ6fc6bvzO2VOyrCeUfSBIbR3NdGoztNKCA6uo0BDtU+7XF913K/RAIiSfHF6VN31ZDvQQIFx+jCu4L3VHaEBB7hkpwNQzZUg4GTp3AXqy8NhPIHtFDPKIscX1ra73DL7r4ycCCJknAW3aUCy1AdkY6xK1U1gEFjRAk5mQpKLQk1FARqXHg0snDnYH7M01L4lX4zZpUyFZMiTAByrC+LVsKFyKdsMiQwQ+Gh73tHDyOx5yoWUyYhFmjlUEmjHqMy2n00MHrFjE7JETNxVTIZBEKKdDsTCtG8jdIf+HKcOXyIE/+mAs8ymZOi/Gn367LhImfaCGISJIZO8UAWlcwbXV39+RemYWSrpAEEUS7bM1UtzQDuHMbAithlHfhg31TOSW60PL9+ZpUwEfCvmwFdXGavHdgnhSUmZ830ft0uEgrqe3wYvL3SZWDTRB5sj4TLr1YDEy0gdzogFjrUwvrDthsWAg0CVQVuUiZK+GQssr7wZWKwsvzUAFqaVNRqzHzBoroRk0ASJwvaG4gnJTodzMZ0OxpSA9nRoBOpAXsADHfO0loETkzjuMcMeX9yfK2WzHKrMDPc0A8d+DwHsFkGbbqw7DNbR2gW3jjhOJOjNzqzn4x7jqh4GM5srPu4BLZHZ85YpJ2XcQ+wZwO0I1aPdrlpcPSAOY1OnAQMuK+XZpd5uh3xiBzhXAFZwEGxki3mfhnS8cYQwsKGMi3sCHkCJGhfnIM7IZB6oP5DbmcdQWVbAUE+YgrGp0hRZV7nfhTQaCvOJA6SYAgcd+MB0wSIkQjriOHOhI54FFmlmVNy8KQ6SGJnO0wtmP6RAWVYKA1ACGFWawjDQD0kiv1K7FEeaLsuVRZZ7Pn02bl6tRIxkETzxzE81R2uuiJJ6zOtUMzpku/TcRTwjuzVXSMnjXrcNDtlHEtYxAIfsIw/rerS8aEH0G9uC6Ff8YLW4EwOI99hgf0pKudujFXaE5Pu+1AprPcTpak3Q+rcaAA==) **Figure: High-level overview of KVM/QEMU virtualization** The following are the prerequisites to enable KVM Hypervisor on Arm CPU-based systems: - The Boot loader starts the Linux kernel in the exception level 2 (EL2) - `CONFIG_KVM` is enabled in the kernel configuration To verify if KVM is available on the device, use the following command to verify for the presence of `/dev/kvm`: ls -l /dev/kvm Copy to clipboard In the Qualcomm Linux default boot flow, the Linux Kernel starts in EL1 while Gunyah assumes the role of the Hypervisor. For instructions on booting the Linux Kernel in EL2 to enable the KVM Hypervisor, see [UEFI](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-4/boot-developer-touchpoints.html#uefi). Consider the following when enabling KVM: - Virtual host extensions (VHE) are enabled by default on all the Qualcomm Linux SoCs that support Arm (v8.1) or later instruction sets. This configuration allows the entire Linux Kernel to run at EL2 with slight modifications. The guest OS kernel and user space run at EL1 and EL0. With VHE, transitions between the guest and host incur less penalty because the guest and host kernels operate at different exception levels. - The Linux Kernel manages all accessible non-secure memory. It does not support protected use cases that require specific isolation from the Linux kernel. The Gunyah hypervisor controls the S2 page tables of the Linux Kernel in the default boot flow. - The peripheral image loading (PIL) services are not supported. Hence, any applications that require aDSP, cDSP, and neural signal processor (NSP) do not work when KVM is enabled. - The power state coordination interface (PSCI) for KVM is supported on the Linux kernel. PSCI ensures that CPU hotplug and low-power modes (LPM) function correctly with KVM. ### Virtual machine manager Quick emulator (QEMU) is used as the virtual machine manager (VMM) for virtualization. Libvirt acts as a management layer that interacts with QEMU to start, stop, and manage virtual machines. ### QEMU Use QEMU as a VMM with the KVM to run VMs at near-native performance. QEMU supports emulated devices, for example, virtio devices, that provide high-performance I/O operations and allows access to USB and peripheral component interconnect (PCI) devices from the VM. The QEMU process and its threads manage a single VM. Invoke QEMU directly to create the VMs. For more information, see [QEMU’s](https://qemu-project.gitlab.io/qemu/index.html). ### Libvirt Libvirt is a suite of tools, including an API library, a daemon (libvirtd), and a command-line utility (virsh), for managing VMs. The virsh utility is useful for managing multiple VMs. For more information, see [virsh](https://www.libvirt.org/manpages/virsh.html). ### Launch guest VM To launch an ARM64-based VM, use QEMU or virsh commands that internally work with libvirt. The commands allow you to configure the VM with CPU, memory, and storage. Specify the number of virtual CPUs and the amount of memory allocated to the VM. Additionally, the VM can boot with a ramdisk (initrd) and a `.ext4` root file system. Note Verify that the guest kernel image (`Image`), root filesystem CPIO (`rootfs.cpio.gz`), and root filesystem image (`rootfs.ext4`) are present in the `/mnt/overlay/guest` directory in the host filesystem before launching the guest. These file paths and formats are examples for reference. ### QEMU Use the following procedures to bring up the VMs with various configurations: ### Boot with ramdisk qemu-system-aarch64 \ -M virt -m 2G \ -kernel /mnt/overlay/guest/Image \ -initrd /mnt/overlay/guest/rootfs.cpio.gz \ -cpu host --enable-kvm -smp 4 -nographic Copy to clipboard ### Boot with rootfs image qemu-system-aarch64 \ -M virt -m 2G \ -kernel /mnt/overlay/guest/Image \ -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \ -append "root=/dev/vda" \ -cpu host --enable-kvm -smp 4 -nographic Copy to clipboard ### Libvirt Libvirt enables you to use QEMU and KVM to create, interact, and manage VMs. ### VM management with virsh virsh provides a range of commands to create, control, and monitor VMs. The following are the virsh commands used for VM management: # Define a new VM domain called initrd_simple virsh define /mnt/overlay/libvirt_initrd_simple.xml Copy to clipboard # List all domains and check the state of initrd_simple, it is not started yet virsh list --all Copy to clipboard # start the initrd_simple VM virsh start initrd_simple Copy to clipboard # connect to console virsh console initrd_simple Copy to clipboard # Disconnect from console by pressing (Ctrl + ]) key combination Copy to clipboard # shutdown the VM Copy to clipboard virsh shutdown initrd_simple Copy to clipboard # Undefine the VM if there is no intention to restart/resue it virsh undefine initrd_simple Copy to clipboard ### Boot with ramdisk Copy the following XML content to `/mnt/overlay/guest/libvirt_initrd_simple.xml` on the host: simple_initrd 2097152 4 /machine hvm /mnt/overlay/guest/Image /mnt/overlay/guest/rootfs.cpio.gz Copy to clipboard ### Boot with rootfs image Copy the following XML file to the host file system in the `/mnt/overlay/guest/libvirt_rootfs_simple.xml` file. Use the VM management commands with `rootfs_simple` as VM domain/name instead of `initrd_simple`. simple_rootfs 2097152 4 /machine hvm /mnt/overlay/guest/Image root=/dev/vda Copy to clipboard ### Virtio framework Virtio abstracts devices in a paravirtualized hypervisor environment. It provides an I/O paravirtualization framework to interact with the paravirtualized (paravirt) devices. The virtual machine monitor (VMM) or hypervisor (HYP) emulates most of the devices exposed to virtual machines (VMs) using virtio. ### Key features The following are the key features of virtual network: - Paravirtualization and full virtualization > > > - Full virtualization: In a fully virtual environment, the guest VM does not recognize the hypervisor, it runs without modifications. However, full virtualization results in higher overhead due to the device emulation. > - Paravirtualization: In a paravirtualize environment, the VM recognizes the hypervisor and requires modifications to the OS. Paravirtualization allows efficient communication between the guest and the host. - Virtio architecture > > > ![../_images/virtio_arch.png](data:image/png;base64,UklGRpgPAABXRUJQVlA4TIwPAAAvWcKGABfkNgDbtiFISWlb/h/yVH5RA/AG47aNHE3agOv/fY1dG3n27HHYtpEj7czG+9zBd//NfD2b1575jxDXZZpH5SpEpIsoryzoQ6R4pjdmACOvqHkCVYjNSiHUIijJENAiAKPqpqqo7qiOrjflAQIs8JxF96ya0JeMlmBM5d5zFs2K1h3eGQEi+PkgQIQQQ0KIIYE4iIMEFxHHxRZiSBDpYnPRxaIyKxsR/sff+iaX2bmIdJ9IXom0cxHJma2cIgaomKdjUrRp30LWiXRfT9adtRGQvKxvEsk+zJMilrM2ApTeZkUA05MO8x0IAJPDZj1kXfR7+d/MtxMpt+Y7v0LFc1yeo4Cp0ZLRM4gYPesRwcRoWYsIKj7bw2//kVjc/r9x2wYd2Ymy905kZTBLzZC6GCt77+06O16p5cqq6baqWjF/Ou+HH0DpGB1gxTf4PRyIEHcR/acESZLgtskJAwEIR/XayztgAXju/RubVe8u3Wc16oGP99K9wnLULXvpRtthure6xmJUl/sv99+FSC+fNf5PRX+ZXqdh/vVranUWwBtp9pPR1/O5//KrPLHaF8DhPy3vsmcXNOIHAKFrZcsuAOivtObOQa6/7C5AZRVydcogC8+YuoCF1WT75W6K/l4KgCPxHRASFaYrTyZBIuDN3CpCK0UoF7C8dzqEkovfjaxhmckUjdwTQMmEqOD4uQqQZG4B01h6mmENCCtJJlpc/GnLxC+BUQg+FIzcKhybovQaygWQqtgMaxo6dIpnjF1yURv+Zn3VgVfHAlae8KUj+8IA13q/HODIjTHiLydPFOcC/bo0eYAf0wrm3M4QkA+5aoAyObNZf75s+JjDqlpfytAiJAfWfkWfxu0CbmENWFu2E8CV9xWZn+7pXcEckFcigNLIBcUTs7H5p6WdX6kWUCruDvzYHKKRW0U3NuXqSJX84KjWoVOlUMQCDH0dbaHVViYCntbIWgJqLabcVTlF6wMqI6e9aO9r2XJiJdGhTzwt+slD6OyXqOIvqNb+Ky7I3NBveB8A8lqgrKXDq48Iyz89sYJEu4B+6TsnUCZTM4wStlG60Y35fmUAVpMgT5v1+5X6H45Yq6Qn/qQq8u+0kjKRSPSR9MDdFwDG+4xF3/w/Y+F9W/cXBNYcM/HKCISn2fcrETvpgBqAtVUm3FTn42V9/nNha/RzYTbrs6y5/3L/5f47mOrx5zJktRnDU79HhqzfVtnCn5EpK2QO1z6fEeuFLCL4jFiz3H95ZPiPsezaGWbs4ojGzgLYBoBFwQWdON4fEAG0a+iPPJmcX4YrmWxlXyAuMwdt7gw01MGkRGsrDFUWZnAk7ROYSqZo+cEw8ZjT2zBAfQHH9LZbqCTllORYlrFnoKEMxmReYSiysINLQvTF00znT34wTDzu9DbMT/uEH10uILa+jZi2UppY+uXDiFJQxRloqINJmacMRRZmcFSFSERUisEw8djT2zAuqi3JborrIQzRUStSx06J4gw0lMHm0hztM9RZAHkXzBx9PaZag6nYX7kWU+PTJg4zx8ZAlWQKMhRnoKEIxp0oS8dQZwFazipq730NpxqMHI89vQ3zu6EKA0gxeHq/CF6izKyOoJHSt1ij5xOhOxju9DZsC3k/WuKw9mD409swZ0z4PVfBeYdKbzDK09swP8q7tQV02vpSomUGm0WLX5gmt/QGozy9DfOj3EyT4gw0lGopUWSwWfQ4rMD5MQbTSq8zDBG/jqblaa84Aw1FsLHX0fSUdGVXdzD86W2YIMX5ZWjoCwB8MiV5sb5isFn0+MPSur7WYBSnt2GEmPcrW6+jjMCUPQMNJSnRWseNiM2iqUJN4ukMhj+9DZv1WdbcfxdtdAGtK8sfLVYnqV+vzJIu/YPN6gWb+y8rucbo67I0ewjmX6+m1lvd/ls3hy5b1NkmZfb1evHqOZpsURnS98DUUnlxLjB/0U75HgCmGdZ+f51c2SjfA+TKRvn2RAEUxdxObdYATdPk/sv9l/sv91/uvwNnANaHh+wVwDLdOBDlJRSSAl6KB/gKP4DWyCvfFDj5ppyJH4f/skAxa5tGlYiNZPQXDm8SHVzDcMCXjPdAqtDRHpIiKfGRTF6BGTFDwal845n94SHMfNM66CjwySjRl6LwKopIBq8p4EmwouG8x2b9XJryJzV8Ij5paLJWRRHJ5LUITdMEfMnyrJPidXqY1uAgP0c8qaKIZPI8KjJvX2E1EkViDeBKRRHJ5DUOrzctTjY2SoEvmw7F3mBGpqAaptFrDhXJLT38cojntYmTotPRjpBIdsCX+ER0YwhwdDlE4hOfJD6Hk5fgVIkUkczeKwC8HrrDqzBrm5dmIGY4+Xrz0ici4Qu4tum+BFpVIkUks9ecFFWjxQEAs4wfiNf5NwIKSCkaVSJFJMNXIOhpZ6Drtp9XqF5pACkMffNSflMSszebRpWIj2TEJjX6AQ5LIDW1Ud6lopW6sUSvrZRIri3Vt9GWkmtLJQIxO3vj+In9tW6JRdaoz9LrxauF8dfXqfUGzF+RZg+eN/n6Z8o9OjT5+u9FFv37fIas/2cM/xBZsq7LGG4XmbKyh8x4311ZxDAr1oO5/wyUkGpjsAsBd3s4Y2cRMGdAzzqRR7DVE4PJApgzbIu77RNlgn9tCODMtjTNfS0WB6z/1b3EzubdovZBDhQ2SFQ50XzA0MgiUM6xQdKWopbUouQHw8YrN+cCvYHxXSwmeoLWVuI/0jM/Rri7DuQn0UCRkmI0XNIBg80i2QA26TA6OjoUolMMhonHZJmb3t1xZCB6YYjQI7N4R+BcOdzuiY5g/kEtunIYNsXdJbF4JiTpPEnky2HpawabRTIfhaCi8IQXUTEYLl5PbJbDZEy12W12bjiyKMKQbjtyTpyR9qLU1t105zYQ3RAk15zoaDqMEklzdIOhziLtCyHAHH0haq3BRPb3rRS12dkI6l1oT2zzSSIQpbk5lCZajwhM1E5KusVgs0hKzjnhhoATm6rBcPFGG8TS+B6yXYu7A51SW4MN+YipnvqN6rdATidhkzLYLBopUZK9cE8E/cFE8qDNwtSojpZbi2Su6rm7J3qpIu9Hg+iNM5iiRx8SzA4EyKFy8Zzb3tETAjBQZt1zcbRVrXgqp5G7jMkPPmOAFkUc4+4hnMAOybOlQbV+0TDYLFpwd53cgt5g2AoCZseJHhNjID2x3yR7K2kzMjs7kqeT17+UnJToDIPNoqcWA9RD6A9mUXqdYXZ2BgIxmUL1EGUtgJL8fd2QX0cD7hZulKdOXsBujdIFxRwdex1NT5BejmgPZiB6xSiSN7QbXNVDwElraovY0lv7RyHu5vIgDtVCTQMx2Cx60KPr+vqD6dEs9Y7JWTyzzbwI3CikYyEaL/j3K+mqrfTWInwxVJISLRbcYZPNoikKR+JpDwZdL3ktY7M+y5r7L/ffLw35rFhXZA53WaxOUuevypB+95jN6gWb+y/33y8GCbARUjCGg1iAgzqCWJS7/JIkbi41xdfskM+30CfBlS3zQ5J0k++pb+wXbDtrJ+rEQG7Io9shn2+hT6ha5kdBqmaTmjumwTR6IibQC0m/sLuHuh3y2Rb6lKJlfim1KuN76ps0je8sB6Y/WaJHwm2IgV6HfEULfYmAfBDn+iZ2bFJzR2+S42cvwR1k9Trk8y30KVXL/J4cW05q/JudS/ZpXgz4UAJ6fcz5FvoU2zKfjy0nNX5B+CRD2A3KFvoQ4FvmWwfSxsnZJelQ51W20IcA3zLfSnCi3kZNNfLdeh3yuRb6GmMIAoq7bYEdgdFPpbx0qHV6bfe5FvoyvmW+YjNbAD0BDClpIWxxqNd2n2uhL+Nb5ivW0awBJ7Ah2aoBoA6aHfLVLfQF+Jb5iqQWAHPTyUnCaMKe0+6Qr26hL8C1zOffryyHmc1+UpyQzkloAzixMbRUdhbFtqUiAD+0VBZFPWTYrG+j5f7L/XcwzOx2k69b0+xVmH6lWXPW+N3yVqpkgHr1kjubLFHZ0mngC0ulukSgtVROA6MrK6W6JNFaKacBcmWjHC9O4/Ki+NJK3d3MyX+53H+5/3L/5f7L/feLQTbr/xZuszocHKzqytKcJtcWq/vdhbGOnTa2y3Ds9uW64bYuO9TS3GKdzUFMrTlw/T0mX3cAJ9Ls76+ZfH2Q+y/3X+6/3H/5sAA2wq6kvJiUI6cAHHtfivL2w2hG3Hf3IxxNfIRjhMM7REBkoupkrY7NLIJ10IrUKeC7uv4UpI4m0SoQBQqatmKj8llnFUh9ag0cxfpsNBvXUVMPu9rVdYVjrnahSnxH52gFJCIKNqoq69zVswKVM3ZcEYEeXB1A0Nl7VDrynkpE8nOPoMGc/NjIRlVknZMYX6GyAjT2oYCjM/l9IgLJ9hWZ+Q6BhOB3u4qsNI3DukXYzMWjpyqZKhzxHd6vP8Wxeh3xtSkKZdQaUP+qWAPxYQDAWCK+qk/h0+QPX6sQmajWQwRCdeR9N5b6/jBLAjjAhYqLqmYd1tGCtDqhcAqReJ+qcIS8lFjHp/iOi6qyjmgVcDmOKhzBKboaS80w2pbEeRiRi6pSSVkjGlsgJHs/VwBwTCrVOhoJh7omN4MqKstVODWrYwHMbIEjIPVVQGQw6/ifMvvBY9KRtlJGZbM2sA1r/3VcB07F5ODJpJLfr4y1ZCZN0yOYKaOqs+JYtFmfZc39l/sv91/uv9x/uf+Mx/Xe5OvXNHsV5l/HU6uJnenX0jxVrEPl/juYa1pYjJruNbtRe+tH2+GhZi/rrY+tRr3bWKzKGTaJX3Na+tLIfYYJqZepLx64b5Lqxxczvq8nxZx4+RIxWfUl5+wlYgLrgVf3f3+rU1+tbI4J4xjvTuoXtTO253+amHqO9zFw1e2TVjcBhSl7dDhpdf5iCUAepw5zYAqAJABqkMcp05AEP/MEkLmRItNX/jN5O5qjRs3k+lknkvsz/AVqbupCAoCJlSpIyoNm4NLoJUwJQbdLLiHvOAWz+xRcGBpgKLhsGY90oYFLUyN9d4ObrZMFdFNuO0ARQz21AZEFQQY6hWkAcJBGAEkAEmGECeMBmkeKg9RcSJsCDGnm8ri5DiDzkWYuatUUZtLQh6QQvAchb8eHkWd5Kmk85RPazGyFFtwDPDttmQAAfw/S8pBduiES7NQ2TSRSqnDzVKiewakfkuRKsA8xPlo3wPBIzwqt2IsfZIB0HiJvaiHsYo3/V6ULrAPYrE/vWHwXsL7ZMRHFeuG5iam/8j7M5N9G+2xCXPLypH/PsXlkIt0yb/b7qxMTUZ81k/nd7M8ak66Pi4kq92JjsSr3X+6/i7NoAA==) > > **Figure : A high-level overview of the virtio architecture** > > - Front-end drivers: Implements in the guest OS. The front-end drivers interact with the back-end drivers > in the hypervisor > - Back-end drivers: Implements in the VMM/HYP. The back-end drivers handle the actual device emulation, > and interacts with the front-end drivers through virtual queues > - Virtual queues: Virtio uses virtual queues (virtqueues) to facilitate communication between front-end and back-end drivers. These queues are implemented as rings to manage guest-to-hypervisor transitions efficiently. - Vhost > > > Vhost is a protocol that offloads the virtio data plane implementation to another element > (user process or kernel module) to enhance performance. This offloading reduces the overhead > of context switching between the guest VM and the hypervisor. - Vhost-net: A kernel-level implementation that allows the data plane to bypass the QEMU process, to reduce the latency and improves the performance. For more information on v-host-net, see [Introduction to virtio-networking and vhost-net](https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net). - Vhost-user: A user-space implementation that handles the data plane with a separate process, to provide flexibility and better performance for workloads. For more information on v-host-user, see [Vhost-user Protocol](https://www.qemu.org/docs/master/interop/vhost-user.html). ### Benefits The following are the advantages of using virtio framework: - Standardization: Virtio provides a common interface for device emulation. It promotes code reuse and efficiency across different virtualization platforms. - Flexibility: Virtio supports block devices and network devices. ### Virtio interfaces The following are the virtio-supported interfaces: ### 9P transport overview `virtio-9p` enables you to share the files between the host and the VMs under the 9P (Plan 9 file system) protocol to enhance the performance within the virtio framework. Note Ensure that the following configurations are enabled in the host and guest kernels: Host side: CONFIG_NET_9P, CONFIG_NET_9P_VIRTIO Copy to clipboard Guest side: CONFIG_NET_9P, CONFIG_NET_9P_VIRTIO, CONFIG_9P_FS Copy to clipboard To create a `virtio-9p` shared directory, define it on the host, and configure the guest VM to mount it. See the following example: > > > root@qcs9100-ride-sx:~# ls -la /mnt/overlay/test_dir > total 12 > drwxr-xr-x. 2 root root 4096 Apr 29 21:36 . > drwxr-xr-x. 7 root root 4096 Apr 29 21:36 .. > -rw-r--r--. 1 root root 8 Apr 29 21:36 file.txt > root@qcs9100-ride-sx:~# cat /mnt/overlay/test_dir/file.txt > testing > root@qcs9100-ride-sx:~# > Copy to clipboard Create a `/mnt/overlay/test_dir` directory on the host to share it with the guest VM. Pass this information either in the libvirt XML configuration or as QEMU arguments. To configure the 9P transport, run the following command: qemu-system-aarch64 \ -M virt -m 2G \ -kernel /mnt/overlay/guest/Image \ -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \ -append "root=/dev/vda" \ -cpu host --enable-kvm -smp 4 -nographic \ -fsdev local,id=fsdev0,path=/mnt/overlay/test_dir,security_model=passthrough \ -device virtio-9p-pci,fsdev=fsdev0,mount_tag=hostshare Copy to clipboard To identify and mount the shared directory on the guest VM, use the `fsdev0` and `hostshare` filesystem devices. To launch the Libvirt guest VM from the libvirt interface with the following sample XML, see [Libvirt](https://docs.qualcomm.com/doc/80-70017-3/topic/features.html#id16). simple_9p 2097152 4 /machine hvm /mnt/overlay/guest/Image root=/dev/vda Copy to clipboard Note Copy the XML content to `/mnt/overlay/guest/libvirt_virtio_9p.xml` file in the host. After the guest VM boots up, verify if the virtio device is probed successfully. > > > root@v8a-arm64:~# lspci > 00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge > 00:01.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 01:00.0 Unclassified device [0002]: Red Hat, Inc. Virtio 1.0 filesystem (rev 01) > 02:00.0 SCSI storage controller: Red Hat, Inc. Virtio 1.0 block device (rev 01) > root@v8a-arm64:~#root@v8a-arm64:~# cat /sys/bus/pci/devices/0000\:01\:00.0/virtio0/uevent > DRIVER=9pnet_virtio > MODALIAS=virtio:d00000009v00001AF4 > root@v8a-arm64:~# > Copy to clipboard To mount the shared directory in the guest VM, run the following command: mount -t 9p -o trans=virtio hostshare mountpoint Copy to clipboard ### VSOCK overview VSOCK is a virtual socket interface that allows communication between VMs and the host OS. It is used in KVM and QEMU. VSOCK is supported through a virtio interface. virtio-vsock is a vhost-based virtio device where the host kernel manages all the data transfer while the KVM hypervisor controls the information. To create a VSOCK connection in a KVM and QEMU environment, specify the context identifier (CID) and port the number. The CID is a unique identifier assigned to each VM in a VSOCK environment, which is used to route communication between the host and the VMs. The host has a CID of 2 while VMs are assigned CIDs starting from 3 and above. The following table lists the CID values: Table: CID values | CID | Description | | --- | --- | | -1 | Any address for binding | | 0 | Hypervisor | | 1 | Loopback | | 2 | Host | Note Ensure that the following configurations are enabled in the host and guest kernels. Host side: CONFIG_VSOCKETS, CONFIG_VHOST_VSOCK Guest side: CONFIG_VSOCKETS, CONFIG_VIRTIO_VSOCKETS Copy to clipboard - Enable VSOCK device: To enable the VSOCK device, specify the device in the VM configuration file (libvirt XML), or pass an argument to QEMU. qemu-system-aarch64 \ -machine virt -m 2G \ -kernel /mnt/overlay/guest/Image \ -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \ -append "root=/dev/vda" \ -cpu host --enable-kvm -smp 4 -nographic \ -device vhost-vsock-pci,guest-cid=73 Copy to clipboard In this command, `guest-cid=73` specifies the CID for the VM. - Launch the guest VM using the libvirt interface: To use the VSOCK device with libvirt, define the VSOCK device in the VM XML configuration file. To launch the guest VM using the libvirt interface, see [Libvirt](https://docs.qualcomm.com/doc/80-70017-3/topic/features.html#id16). Copy the XML content to `/mnt/overlay/guest/libvirt_virtio_vsock.xml` on the host. After the Guest VM boots up, verify that the device is probed successfully. > > > root@v8a-arm64:~# lspci > 00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge > 00:01.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 01:00.0 SCSI storage controller: Red Hat, Inc. Virtio 1.0 block device (rev 01) > 02:00.0 Communication controller: Red Hat, Inc. Virtio 1.0 socket (rev 01) > root@v8a-arm64:~# > root@v8a-arm64:~# cat /sys/bus/pci/devices/0000\:02\:00.0/virtio1/uevent > DRIVER=vmw_vsock_virtio_transport > MODALIAS=virtio:d00000013v00001AF4 > root@v8a-arm64:~# > Copy to clipboard To verify the host-to-guest and guest-to-host interaction, use the `socat` utility. Ensure that the `socat` utility is supported on host and guest. - On the host, verify the following port: > > > socat STDIN VSOCK-LISTEN:1234 > Copy to clipboard - On the guest, connect to the host using the host CID (2): socat STDOUT VSOCK-CONNECT:2:1234 Copy to clipboard - In this setup, the guest and host show the same values. ### Virtio block overview The virtio block is a standardized way to present block devices to the VM. Each virtio block device appears as a disk inside the guest VM. The virtio block allows the VMs to read and write operations. The following are the key features of the virtio-block device: - Simplicity: Easy to implement and use - Performance: Designed to minimize overhead and maximize throughput - Flexibility: Designed to use with various types of storage backends Note Ensure that `CONFIG_VIRTIO_BLK` is enabled in the guest kernel. To use a virtio-block device, do the following: 1. To configure the VM for including a virtio-block device, specify the device in the VM configuration file (libvirt XML), or pass an argument to QEMU. 1. To enable the virtio-block device in the VM, use the following QEMU command: qemu-system-aarch64 \ -M virt -m 2G \ -kernel /mnt/overlay/guest/Image \ -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \ -append "root=/dev/vda" \ -cpu host --enable-kvm -smp 4 -nographic \ -drive file=/mnt/overlay/guest/disk.img,if=virtio,format=raw Copy to clipboard In this command, `rootfs.ext4` and `disk.img` are the disks inside the Guest VM. 2. Launch the guest VM using the libvirt interface: To launch the guest VM with the following sample XML, see [Libvirt](https://docs.qualcomm.com/doc/80-70017-3/topic/features.html#id16): simple_block_device 2097152 4 /machine hvm /mnt/overlay/guest/Image root=/dev/vda Copy to clipboard Note Copy the XML content to `/mnt/overlay/guest/libvirt_virtio_blk.xml` on the host. 2. The guest OS detects the virtio-block device and initializes it. In this process, the guest driver communicates with the device to set up the necessary data structures and queues. > > > root@v8a-arm64:~# dmesg | grep virtio > [ 0.225425] virtio-pci 0000:01:00.0: enabling device (0000 -> 0002) > [ 0.227073] virtio-pci 0000:02:00.0: enabling device (0000 -> 0002) > [ 0.234012] virtio_blk virtio0: 4/0/0 default/read/poll queues > [ 0.235405] virtio_blk virtio0: [vda] 2220734 512-byte logical blocks (1.14 GB/1.06 GiB) > [ 0.241213] virtio_blk virtio1: 4/0/0 default/read/poll queues > [ 0.242418] virtio_blk virtio1: [vdb] 8192000 512-byte logical blocks (4.19 GB/3.91 GiB) > root@v8a-arm64:~# > root@v8a-arm64:~# lspci > 00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge > 00:01.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 01:00.0 SCSI storage controller: Red Hat, Inc. Virtio 1.0 block device (rev 01) > 02:00.0 SCSI storage controller: Red Hat, Inc. Virtio 1.0 block device (rev 01) > root@v8a-arm64:~# > Copy to clipboard 3. After initializing, you can use the virtio-block and other block devices. The guest OS performs read and write operations on the virtio driver and passes the operations to the host for processing. 4. To manage the virtio-block device, use the standard tools and commands available in the guest OS. For example, `lsblk` and `df`. ### Virtio-IOMMU overview Virtio-IOMMU is a paravirtualized input/output memory management unit (IOMMU) that provides direct memory access (DMA) management in the virtual environments. Virtio-IOMMU integrates with the existing software APIs such as virtual function I/O (VFIO) and removes the need for page table emulation, making it a lightweight and high-performance solution. Virtio-IOMMU acts as a proxy for physical IOMMUs and manages the devices assigned to the guest. As virtual IOMMU, it manages, emulates, and paravirtualizes the devices. The following are the key features of Virtio-IOMMU: - Paravirtualization: Leverages existing transport mechanisms and reduces overhead. - Flexibility: Supports PCI passthrough and shares virtual memory. - Integration: Integrates with software APIs and enhances compatibility. For example, VFIO. Note Ensure that `CONFIG_VIRTIO_IOMMU` is enabled in the guest kernel. To use a Virtio-IOMMU device, do the following: 1. To configure the VM for including a Virtio-IOMMU device, specify the device in the VM configuration file (XML), or pass an argument to QEMU. > > > 1. Enable a Virtio-IOMMU device: To enable the Virtio-IOMMU device in the VM, use the following QEMU command: > > > > > > > > > > qemu-system-aarch64 \ > > -M virt -m 2G \ > > -kernel /mnt/overlay/guest/Image \ > > -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \ > > -append "root=/dev/vda" \ > > -cpu host --enable-kvm -smp 4 -nographic \ > > -device virtio-iommu-pci > > Copy to clipboard > > > 2. Launch guest VM using the libvirt interface: To launch the guest VM with the following sample XML, see [Libvirt](https://docs.qualcomm.com/doc/80-70017-3/topic/features.html#id16). > > > > > > > > > > > > simple_iommu > > 2097152 > > 4 > > > > /machine > > > > > > hvm > > /mnt/overlay/guest/Image > > root=/dev/vda > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Copy to clipboard > > > > Note > > > Copy the XML content to `/mnt/overlay/guest/libvirt_virtio_iommu.xml` on the host. 2. The guest OS detects the Virtio-IOMMU device and initializes it. In this process, the guest driver communicates with the device to set up the necessary data structures and mappings. > > > root@v8a-arm64:~# dmesg | grep virtio > [ 0.184166] virtio-pci 0000:00:03.0: enabling device (0000 -> 0002) > [ 0.189122] virtio_iommu virtio0: input address: 64 bits > [ 0.189516] virtio_iommu virtio0: page mask: 0xfffffffffffff000 > [ 0.220135] virtio-pci 0000:01:00.0: Adding to iommu group 0 > [ 0.220676] virtio-pci 0000:01:00.0: enabling device (0000 -> 0002) > [ 0.223065] virtio_blk virtio1: 4/0/0 default/read/poll queues > [ 0.224713] virtio_blk virtio1: [vda] 2220734 512-byte logical blocks (1.14 GB/1.06 GiB) > root@v8a-arm64:~# > Copy to clipboard > > > The DMESG logs show that the virtio-block device is attached to the IOMMU domain > which is a part of the iommu group `0`. 3. After initializing, the Virtio-IOMMU device manages DMA operations for attached devices. The guest OS performs mapping and unmapping operations with the virtio driver and passes the operations to the host for processing. > > > The following example shows Virtio-IOMMU device operations: > > > root@v8a-arm64:~# cat /sys/kernel/iommu_groups/0/devices/0000\:01\:00.0/virtio1/uevent > DRIVER=virtio_blk > MODALIAS=virtio:d00000002v00001AF4 > root@v8a-arm64:~# > Copy to clipboard ### Virtio-net overview The virtio-net device is a virtual network device that provides an interface for network operations in virtualized environments. It offers high performance and low overhead. The following are the key features of the virtio-net device: - Efficiency: Minimizes overhead and maximizes throughput. - Simplicity: Easy to implement and use. - Flexibility: Supports various network configurations and backends. Note Ensure that `CONFIG_VIRTIO_NET` is enabled in the guest kernel. To use a virtio-net device, follow the steps: 1. To configure the VM for including a virtio-net device, specify the device in the VM configuration file (libvirt XML), or pass an argument to QEMU. > > > 1. To enable the virtio-net device in the VM, use the following QEMU command: > > > > > > > > > > qemu-system-aarch64 \ > > -M virt -m 2G \ > > -kernel /mnt/overlay/guest/Image \ > > -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \ > > -append "root=/dev/vda" \ > > -cpu host --enable-kvm -smp 4 -nographic \ > > -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \ > > -device virtio-net-pci,netdev=net0 > > Copy to clipboard > > > > > > Note > > > > > > This command creates a Tap interface on the host. To create the Tap interface manually, > > use the following command: > > > > > > ip tuntap add dev tap0 mode tap > > Copy to clipboard > > > 2. To launch the guest VM with the following sample XML, > > > > > > > > > > see [Libvirt](https://docs.qualcomm.com/doc/80-70017-3/topic/features.html#id16). > > > > > > > > simple_net > > 2097152 > > 4 > > > > /machine > > > > > > hvm > > /mnt/overlay/guest/Image > > root=/dev/vda > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Copy to clipboard > > > > Note > > > To create a tap interface for direct communication between the host VM and the guest VM. > Copy the XML content to `/mnt/overlay/guest/libvirt_virtio_net.xml` on the host. 1. The guest operating system detects the virtio-net device and initializes it. In this process, the guest driver communicates with the device to set up the necessary data structures and queues. > > > root@v8a-arm64:~# dmesg | grep virtio > [ 0.394365] virtio-pci 0000:01:00.0: enabling device (0000 -> 0002) > [ 0.396364] virtio-pci 0000:02:00.0: enabling device (0000 -> 0002) > [ 0.404496] virtio_blk virtio1: 4/0/0 default/read/poll queues > [ 0.406048] virtio_blk virtio1: [vda] 2220734 512-byte logical blocks (1.14 GB/1.06 GiB) > [ 0.980919] virtio_net virtio0 enp1s0: renamed from eth0 > root@v8a-arm64:~# lspci > 00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge > 00:01.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 01:00.0 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01) > 02:00.0 SCSI storage controller: Red Hat, Inc. Virtio 1.0 block device (rev 01) > root@v8a-arm64:~# > Copy to clipboard 3. Operate: After you initialize and configure the virtio-net device, use it as other network interface. The guest OS performs standard network operations with the virtio driver and passes the operations to the host for processing. To use the Tap interface, run the following network configurations on the host VM and the guest VM: - Host configuration: root@qcs9100-ride-sx:~# ip addr add 192.168.100.1/24 dev tap0 root@qcs9100-ride-sx:~# ip link set dev tap0 up Copy to clipboard - Guest configuration: root@v8a-arm64:~# ip addr add 192.168.100.2/24 dev enp1s0 root@v8a-arm64:~# ip link set dev enp1s0 up Copy to clipboard To run the network traffic on this interface, use the following example: > > > root@v8a-arm64:~# ping 192.168.100.1 > PING 192.168.100.1 (192.168.100.1): 56 data bytes > 64 bytes from 192.168.100.1: seq=0 ttl=64 time=0.364 ms > 64 bytes from 192.168.100.1: seq=1 ttl=64 time=0.156 ms > Copy to clipboard ### Virtio-serial and virtio-console overview The virtio-serial device that provides an interface for serial communication between the host VM and the guest VM in virtual environments. The following are the key features of the virtio-serial device: - Multiple ports: Supports multiple serial ports, enables various communication channels - Efficiency: Minimizes overhead and maximizes throughput - Flexibility: Supports different types of data exchange. For example, console access and file transfer Note Ensure that `CONFIG_VIRTIO_CONSOLE` is enabled in the guest kernel. To use a virtio-serial device, follow the steps: 1. To configure the VM, for including a virtio-serial device, specify the device in the VM configuration file (libvirt XML), or pass an argument to QEMU. > > > 1. To enable the virtio-net device in the VM, use the following QEMU command: > > > > > > > > > > qemu-system-aarch64 \ > > -M virt -m 2G \ > > -kernel /mnt/overlay/guest/Image \ > > -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \ > > -append "root=/dev/vda" \ > > -cpu host --enable-kvm -smp 4 -nographic \ > > -device virtio-serial-pci,id=virtio-serial0 \ > > -chardev pty,id=charconsole0 \ > > -device virtconsole,chardev=charconsole0,id=console0 \ > > -chardev socket,path=/tmp/qemu.sock,server=on,wait=off,id=charchannel0 \ > > -device virtserialport,chardev=charchannel0,name=org.qemu.guest_agent.0 > > Copy to clipboard > > > 2. To launch the guest VM with the following sample XML, see [Libvirt](https://docs.qualcomm.com/doc/80-70017-3/topic/features.html#id16). > > > > > simple_serial > 2097152 > 4 > > /machine > > > hvm > /mnt/overlay/guest/Image > root=/dev/vda > > > > > > > > > > > > > > > > > > > > > > >
> > > > Copy to clipboard > > > Note > > > Copy the XML content to `/mnt/overlay/guest/virtio_serial.xml` on the host. 2. The guest operating system detects the virtio-serial device and initializes it. In this process, the guest driver communicates with the device to set up the necessary data structures and ports. > > > root@v8a-arm64:~# lspci > 00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge > 00:01.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 01:00.0 Communication controller: Red Hat, Inc. Virtio 1.0 console (rev 01) > 02:00.0 SCSI storage controller: Red Hat, Inc. Virtio 1.0 block device (rev 01) > root@v8a-arm64:~# > root@v8a-arm64:~# cat /sys/bus/virtio/devices/virtio0/uevent > DRIVER=virtio_console > MODALIAS=virtio:d00000003v00001AF4 > root@v8a-arm64:~# > Copy to clipboard 3. After initialization, you can use the virtio-serial as a serial device. The guest OS performs read and write operations with the virtio driver and passes the operations to the host for processing. 4. To manage the virtio-serial device, do the following: - Monitor the performance - Verify the device status - Perform maintenance tasks - Use the standard tools and commands available in the guest OS ### Memory balloon device The traditional virtio memory balloon device manages the guest memory. It allows the host system to reclaim memory from VMs. The memory balloon instructs VMs to return a portion of their memory to the host. This process involves inflating the memory balloon inside the Guest VM, which reduces the memory available for other tasks within the VM. The Guest OS decides which memory pages to give back to the host, indicating which pages it does not need or access. The host then un-maps these pages from the guest VM and marks them as unavailable for the guest VM and allows the host system to use them. If the guest VM requires more memory later, the host deflates the balloon to return the pages. This feature allows each guest VM to continue running while its available memory is managed. Use the following for the virtio-balloon device to relocate physical memory between a guest VM and the host: 1. Balloon inflation: The guest driver allocates memory and informs the host. The host then reuses the inflated memory for other VMs. 2. Balloon deflation: After informing the host, the guest driver frees the previously allocated memory and enables the guest VM to use the deflated memory. `Target balloon size` controls the balloon inflation or deflation through a request to change the guest VM memory size. The resize request is sent through QEMU monitor mode or through virsh commands. The following are the key features of the virtio-balloon device: - Memory management: Adjusts memory allocation in real time based on the requirements. - Efficiency: Reclaims unused memory from idle VMs and redistributes it to the active ones. - Flexibility: Enables various virtualization setups to optimize resource usage. Note Ensure that `CONFIG_VIRTIO_BALLOON` is enabled in the guest kernel. To use a virtio-balloon device, follow the steps: 1. Configure the VM for including a virtio-balloon device, specify the device in the VM configuration file (libvirt XML), or pass an argument to QEMU. 1. To enable the virtio-balloon device in the VM, use the following QEMU command: qemu-system-aarch64 \ -M virt -m 2G \ -kernel /mnt/overlay/guest/Image \ -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \ -append "root=/dev/vda" \ -cpu host --enable-kvm -smp 4 -nographic \ -device virtio-balloon-pci,id=balloon0 Copy to clipboard 2. To launch the guest VM with the following sample XML, see [Libvirt](https://docs.qualcomm.com/doc/80-70017-3/topic/features.html#id16): simple_balloon 2097152 4 /machine hvm /mnt/overlay/guest/Image root=/dev/vda Copy to clipboard Note Copy the XML content to `/mnt/overlay/guest/libvirt_virtio_balloon.xml` file on the host. 2. The guest OS detects the virtio-balloon device and initializes it. In this process, the guest driver communicates with the device to set up the necessary data structures and memory management mechanisms. > > > root@v8a-arm64:~# lspci > 00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge > 00:01.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:01.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 01:00.0 SCSI storage controller: Red Hat, Inc. Virtio 1.0 block device (rev 01) > 02:00.0 Unclassified device [00ff]: Red Hat, Inc. Virtio 1.0 memory balloon (rev 01) > root@v8a-arm64:~# > Copy to clipboard 3. After initializing, the virtio-balloon device adjusts the memory allocated to the VM. The guest OS performs memory inflation and deflation operations, which are handled by the virtio driver, and passes it to the host for processing. To adjust the guest memory, run the following virsh commands on the host: - To verify the guest VM initial memory (2 GB), use the following command: > > > root@v8a-arm64:~# cat /proc/meminfo | grep Mem > MemTotal: 1966180 kB > MemFree: 1807432 kB > MemAvailable: 1817552 kB > root@v8a-arm64:~# > Copy to clipboard - To inflate the balloon and reduce the guest VM memory to 512 MB, use the following command: root@qcs9100-ride-sx:~# virsh setmem simple_balloon 512M --live Copy to clipboard - To verify the guest VM memory after inflation, use the following command: root@v8a-arm64:~# cat /proc/meminfo | grep Mem MemTotal: 393316 kB MemFree: 235708 kB MemAvailable: 245836 kB root@v8a-arm64:~# Copy to clipboard - To deflate the balloon (Guest VM memory back to 2 GB): root@qcs9100-ride-sx:~# virsh setmem simple_balloon 2G --live Copy to clipboard - To verify the guest VM memory after deflation: root@v8a-arm64:~# cat /proc/meminfo | grep Mem MemTotal: 1966180 kB MemFree: 1811844 kB MemAvailable: 1821988 kB root@v8a-arm64:~# Copy to clipboard ### Host devices access You can use the enumerated devices on the host system from the guest. The device assignment interface to the guest varies based on the device type. Some devices act as a backend for a device emulated by QEMU, for example, UART. Configure universal asynchronous receiver transmitter (UART) as a character device (chardev) backend on the host. For more information on QEMU chardev, see [Character device options](https://www.qemu.org/docs/master/system/invocation.html#hxtool-6). PCI and USB devices work on the kernel VFIO framework. - PCI passthrough allows a guest VM to directly access a physical PCI device on the host. The VFIO framework in the Linux kernel is used for the device assignment. For more information on the VFIO, see [VFIO - “Virtual Function I/O”](https://docs.kernel.org/driver-api/vfio.html). - A USB device operates in Passthrough mode, allowing the guest VM to use USB peripherals connected to the host. QEMU emulates the USB controller and works with libusb on the host to present USB peripherals to the guest. For more information, see [USB emulation](https://qemu-project.gitlab.io/qemu/system/devices/usb.html). ### USB passthrough overview Identify the USB device that is enumerated on the host computer to assign it to the guest. To see all the USB devices that are enumerated on the host, use the `lusb` command. Find the device by its `vendor id` and `product id`. In the following example, `vendor id` is 0x0781 and `product id` is 0x5567 for the USB memory stick. sh-5.1# lsusb Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 004: ID 0781:5567 SanDisk Corp. Cruzer Blade Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Copy to clipboard Note Copy the following XML file on the host file system in `/mnt/overlay/guest/libvirt_usb.xml` file and modify the XML file as per `vendor id` and `product id`. Launch a guest VM with the following sample XML. After the guest VM is launched, connect to the console, and verify the `lsusb` output in the guest. Ensure that the output lists the USB device assigned to the guest. simple_usb 2097152 4 /machine hvm /mnt/overlay/guest/Image root=/dev/vda Copy to clipboard ### PCI overview Identify the PCI device that is enumerated on the host computer to assign it to the guest. To see all the PCI devices that are enumerated on the host, use the `lspci` command. Find the device by its `[domain:]bus:device.function`. For more information on `lspci` command, see [lspci(8) — Linux manual page](https://man7.org/linux/man-pages/man8/lspci.8.html). The following example shows supported values for domain, bus, device, and function for `Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz` device: - Domain is 0001 - Bus is 01 - Device is 00 - Function is 0 for sh-5.1# lspci 0000:00:00.0 PCI bridge: Qualcomm Device 0115 0000:01:00.0 Network controller: Qualcomm QCNFA765 Wireless Network Adapter (rev 01) 0001:00:00.0 PCI bridge: Qualcomm Device 0115 0001:01:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Copy to clipboard Note Copy the XML content to `/mnt/overlay/guest/libvirt_pci.xml` file and modify the `hostdev` section in the XML file as per your device requirements. To launch the guest VM, see the [Libvirt](https://docs.qualcomm.com/doc/80-70017-3/topic/features.html#id16). After you launch the guest VM, connect it to the console and verify the `lspci` output in the guest. Ensure that the output lists the PCI device assigned to the guest. Note The output shows different slots based on the guest PCI topology selected by QEMU and Libvirt. When `managed` is marked as *yes* in the `hostdev` in the following XML, the PCI device gets detached from the host before it passes on to the guest, and re-attached to the host after the guest exits. For information, see [USB / PCI / SCSI devices](https://libvirt.org/formatdomain.html#usb-pci-scsi-devices). simple_pci 2097152 4 /machine hvm /mnt/overlay/guest/Image root=/dev/vda
Copy to clipboard ### UART passthrough overview Identify the `/dev/ttyX` teletype (TTY) device file corresponding to the UART to use it from the guest. Ensure that no application on the host is using this interface. Note Copy the XML content to `/mnt/overlay/guest/libvirt_uart.xml` file on the host and modify the TTY device file as per your device requirements. Launch a guest VM and connect to the console. The serial interface at `/dev/virtio-ports/hostserial` acts as a front end to the real UART on the host. Use the following XML file: simple_uart 2097152 4 /machine hvm /mnt/overlay/guest/Image root=/dev/vda
Copy to clipboard ### KVM customizations The following features allow you to modify the KVM hypervisor to meet specific requirements: ### Remote command execution To enable the QEMU guest agent in the guest OS user space, run the remote commands on the guest VMs from the host. For more information on `qemu-ga`, see [QEMU Guest Agent](https://qemu-project.gitlab.io/qemu/interop/qemu-ga.html). `qemu-agent-command` is the virsh command that includes `guest-exec` and `guest-exec-status` as the subcommands to execute and verify the status or output of the guest VMs. For more information on how to configure `qemu-ga` through virtio-serial interface, see [QEMU Guest Agent](https://wiki.libvirt.org/Qemu_guest_agent.html). For more information on how `virt-exec` wraps virsh commands and provides an interface, see [kvm-qemu/virt-exec](https://github.com/ildar-shaimordanov/my-scripts/blob/master/kvm-qemu/virt-exec). The following example shows how to run `/proc/meminfo` on the guest VM from the host computer using a `virt-exec` utility. sh-5.2# ./virt-exec testvm cat /proc/meminfo | grep -i total MemTotal: 1968056 kB SwapTotal: 0 kB VmallocTotal: 133141626880 kB CmaTotal: 32768 kB HugePages_Total: 0 Copy to clipboard ### Watchdog configuration A QEMU-emulated I6300 ESB watchdog device is supported in the guest VM. The Linux kernel has a driver for this watchdog and exposes the standard watchdog character device. To enable the watchdog, set `CONFIG_I6300ESB_WDT` in the guest kernel configuration. A guest userspace daemon must pet the watchdog. The guest Libvirt XML has an option to select `poweroff` or `reset` on watchdog timeout. For more information, see [Watchdog devices](https://libvirt.org/formatdomain.html#watchdog-devices). Add the following snippet to the guest XML to enable I6300 ESB watchdog emulation in QEMU. The default action is `reset`, which restarts the guest. Copy to clipboard ### KVM traces To enable KVM traces, run the following command: echo 1 > /sys/kernel/tracing/events/kvm/enable Copy to clipboard The KVM guest interactions are recorded in kernel function trace (ftrace). To inspect interactions between KVM and guest VMs, read the kernel trace buffer. QEMU also has support trace events, which can be redirected to kernel ftrace. For more information on how to redirect QEMU trace backend events to the ftrace buffer, see [Trace backends](https://qemu-project.gitlab.io/qemu/devel/tracing.html#trace-backends). This option is selected by default in the Qualcomm Linux release. Use the following command to launch a guest VM with virtio traces enabled in the QEMU code: qemu-system-aarch64 \ -M virt -m 2G \ -kernel /mnt/overlay/guest/Image \ -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \ -append "root=/dev/vda" \ -cpu host --enable-kvm -smp 4 -nographic \ -trace "virtio*" Copy to clipboard ## Real-time (RT) kernel overview A real-time system is a deterministic system, where response to an event is expected in a set time. A system is classified as compatible with RT if: - It is devoid of unbounded latency. - The maximum response time is calculated with precision. - It meets the set criteria for scheduling of tasks (latency and deadline). Linux can be considered as a real-time operating system (RTOS) in which real-time tasks have well-defined periodic execution cycles (cycle time) and meet execution criteria within specified limits (jitter). When considering Linux as RTOS, apply `PREEMPT_RT` patches and enable configurations for a fully preemptible kernel. The RT kernel maintainers maintain out-of-tree `PREEMPT_RT` patches for every kernel version released. Video: Explore Real-time Linux® Video: Explore Real-time Linux® To install the patches, see [Versions of PREEMPT_RT patches](https://wiki.linuxfoundation.org/realtime/start). Note The real-time support is for kernel space process and not for user space. ![../_images/RT-kernel-build-sequence.png](data:image/png;base64,UklGRuINAABXRUJQVlA4TNYNAAAv8gMlAO8FubatZc15hjOiBPcxhdAjrdCARHv4R8n7832LWBOQa9t6lpwXvo84YgQ6JEMDNGSRLhugBGMFOfv+Z/jg1rbdqDpPmsEjCmCReYgIaYKQWmnEuw68+f8Jozf/fQB+N1PSPP6/vvmp6YhkSppH+6ppIFMy+W5JTfGVBn5KGvCkn+naS9tF+JmayFAZKkNlqPzMYQ8qRywwA8wBHdxhCpgD2rgDt3gHb4AbwN4r4BLQ1kigKtyhKqIKc6PmmMHMUTdmjqkxd9SMuaOSmDuiibsnlrh7KokXT9R48USSKB6aRAF4yJ82o/G44bt53fvcFLDA5+lLhspQH9N/h9FYVMFfnUpRx+rCA5Ui0yLsIWga1CaTAtOisCAsCAtMi1SLTApMi8IClSLToqiiUwUCL0IEUYWIiCq8QFS4Q1VMru5g7w5gB1auoD7V+KSFI2AFFjgAjEpRSfMtf+rv1z/bbK7Y4wjYYAtYAia4AYysAUuwwBwwB+d4ZYEF7GImQ2Wototq5ur/Q4b6bqbtogyVoZqXq5xHMHJiYA6gA9qgbbbBdDl3fj6pv1+XszvouB6mLkxwBIzA4vZ/jptG4wzVdNK7yHnTE9M7Mr2DYm167z30LsrdCV8/kKwIrliH0nuzlBwL0j+385vfjGZmd2ZXBkf0X6JkO1Ftrf0G9Q4Vc8AgIXxATzdfW+Yr1pIBxdowR7FmRpZeJS5JkxnYE+uaL6stseD8+ao1oFpPzFGgXvVMU3EvMxIjNtKAozl/mYG3vjcOnrfOwqFsx1rdr1j7Ehvu4fwT8xVr1YBibZujWL2G0U+jrSk5xVqb7ViHQkxy/tSccm3PKtfqfvVyXWyhg/MxPSGHXdSvBgs6XxdLqZxm5dRqa1ax+hXrgJsNsbakHdhFh3RxNKtUi/vV6mgXjPh3/uycWm3Oymlf2o4jOP+6+qxwA1al7CxzhF7W7eW4M7DODVjnLiTVmOJozp9/7TwrLcO+7ghZy7Bn3H+uHWexfdjkaCxKsKQZC97GhHGgO8Jy2/QyZ2BOyjFjNu4/1y9joSOscU27cZTlnd1e9roBe52Bneo5kczYo42kGljVczhRs9I+bEsJmMhI0IE1JWB6l4S13Xg2j/vPGCTsuLzT0TiSYEkzAutYMKa7AzO78fRec8DGbjybr3OI958rhAw95gvI/Ji3GnXCKZClr8wb5FWKvrzVvx8EJhJO7Xx6ilchQKIuVW+7EKSdT2z0lbnW5FXw1iJoaxcICXxhY1TcnYu+MRwxjxopDlI6r0IaccoRLch6VQ3SVnsXSAFrVknrpE4ayONDlhoIRDHBYfOot5qYI+cNs3EyGHGI+pFXCULTCrUSOds0bV9BTalwM0DcevcV0nqcMXKFhBZAmIcHIfzHIbAaWGcOE92bTCkA1EiDSgizyFrUSAH9BEioVlOeMyVglmENLAYgQA3kRaHBGIIeBiAfh4hnqsQcprsDM9RzUJvzq+08/5v4SAI6V520B1kzqrI3xP+RBowoPPANRMJQLFzxJsUd9AJpnXlkLuwyrVFK7ykFsDPKQALfu0QCG0CgWk1G4IeGBKwVDYK4wyAngYcByMchc9k96VXPHm3USIEfMCCRJGSoRIo+wq/ArFqryQHbpoyhSLjCTIo76H0XS4S0RpmfCpSCr0lT9OGD2TK088hg8ea7/1UiAOjrdp6BfRwCWqd8msQQNJUaaeAB1quQAna5wXWxatHnCU1syhiKhSvMJH3kogJz1j0/hE6rIT1QmoCbdIgF1uq/LHYrHxkgBc7DDOzjQK0eKZ32aWID6T31oo8HWJY3wGHfB2aKeKph05MxFAtXqEnxB+ffmOPpp0S9zED6yuTMow/9rqwGmGHxzWKmWwOblldhzYtvVrCR4bCexp9zaWAZQ/G2i5oUITSmRsHgKVNqSHXGKvfoRIfZ0WTC+S+fMhzmdeRyQPGThpzV4XeaRa9Lg/Szq83OgMaF5zyt0bDP2QBaJafZxOZICcyPPcyWBrDLrSZm2OPgLQmdTIvoyTfkuS+fMpzAZxO9RZ95uEC5b8E/DgF95XY+GbBDF2EAAecUiyQy1Mgv4TkPaJu4oWJQk+KNvjIyXOYfjgUwNVYAcMacHQ74lFk7H1VuJyQ8eFIzMnP9CPzyR8gZ6GHQ7JCPQ/igqpMgGZDThVe5/DFoSVgkkcGr/KnVRKFV0kANFYOYZNplU9f2t04PgkfuMCFDo8yvw9gyx/+WyOn74eVaiY1FospThOn8Z0+aCkzlnn04fH/iFjv9kUvrIh+HEK9CGkkOWuMmhbBIIgOth5EVp6+Mhx4JEJMM/Ccykl72ZBrvvPLSX7+X13efoSKEP4ilXub0aJ89QyKJHJlSIOp2dVJADJUBMck41tuFG58nGnT+Dx+mMGgcgjSoRpnEZsv886IOyKvvWJvbP5mAsDekwzrs0U64QMjl4Q7DXyeVyqKqReUNkovF8cm88MNxUT588/UvPpPQV9/LiOf72y0eWCecP4cc7JEGWctETnUSM0MqmiDtvLHMVsctT0UF4N8CjSIOMYx8mj+yu/O/IeTcbxjyPeTUXErvHSYNncPkjiqVBQ8Mk8ip4+kwSRwRvAo5RaPIZIfgLv7I7txBQv3ztw6pkwL3a4s+l/8YpBQtMwO1Kg+AHj9SIq3HKgE288VXpfGFyjKVytJ4bL2EuLjtQYnaNiPo+TPLF0XlMo0kPQ7BN/DIVrz/zp23c+qQWwmBB/JwfR7oZjaQ4EvKkDIzFP41AYWvSkMKlSUqlQWIatskitvul6ltM4JPYb4oATPhOXZkLV4mKsUfiJEc3nsVMkppnRR9pMyMO/FC+sqth+m8ihD2MMEKlWUqlXEEtW3i4jbJ2jYjuINPEidebr3dZs6/iWjRC+/u2d4haD672mry1ck4NVIAtuEA+9BSNCpRqYwiqG2TKG5TXduWe+ull/7yfWf6/DOor/kk8acTnJC+cmDXVLjOiS/TnP+aDv7+dtjLlNBg1FpNfp0KTp0bEkBwC9CqRXGlMo6wtk1YzqK60GX3s0SDzr35UXrLng/Cy2r0f/7oLjCW7vTyf08PGoG4UhlHUNumnV3PEx165T1NGLNSIvMjrP5IykQ3XcIfwdfnDn6MLzHkTzAkT5DSbA5QiiZdqSxHAN5ccHMZqmvbch+8Ea4Mkdc332P6G8+3t+o7MWWlRL0diwDPgNaI17BgOwYCvApz4UVCWqMU2U9XsPUsHeG32OWSSHGOhi0UI87nBOZLODpeD1BAv4VH8lvQ0mxoH1qoLFGpjCGubRNWsKmubcspf96Sb57u6TGRhpbib9PZr7+XgUFklQQUzOTURMkcH8nzeBWY1cH3060jmZ4aMtjkk0hR25DBKHRsoRhtzoEEKUKnq/7AU4V3lgT4L0JKs3n4F4AClalUlgNWxwgr2CRr28z4RfyRBtAoqXGlBBJw4f7TzC65e+BitRHs0U+mxB4fpJ1nL5aC3FEBz/NkSqf8cJSI7Kcr2HoWmDYiSCLpY3b00bGFoonk1HEHS5AqgM67WELW9mdgJ5AB6VA/lwOkNJt3KlKoLFepLAV8jxFWsEnWthnBP+CRHpAoqXOlBB9wUaTuAYnVkUbwwnf1360m2zJUlDsS7SU49NODoZ3ofrr81rNIpofiSSR9vSzy6NlCMcqEa90EGOP0QnxMkt+ghtuwhb6T1INESX0rJbA919HLUveAxOpYgD1hau1fyw3WwkQTyYI8T5WEOj3Iz6jVW01+oIrNAeEzynEO5m5l+6lFmR7TyJQu+9SrkkaM5MUUgZXz9KqHRUndKyX4gIsjdQ9IrFbNVgPhnivVIHRd+LwQIsjz0HuG2egqj8FvPWsfMHdbF37QW/RjBFXSCdZl9KVNuYArQOoekFitms2Gkin9UinQevFSO488YlECdBR038ekge2nCx2aKYlO+MgU52Duxv0oBjXJGvBTW6d86lIge653DBarYwGvMnS1GbrlSkCFuSM8z1MjJ33qXWo1+f108a1nwYtYJnwGoUmkuEbTForxgUEL3ixLAaJ3pYT4stQ9YLE6FqB1An6eNPDcEQ4bUCBjTWQ/XWQWtOhzg+sGnkSKHDtMQdMWiskOJEpqWymBBlwwsRYieQ9YrI4HagT8fDsvzh3heR5+rIntp4tuPQuTSqMUSyLFdi+TSG8Jc1iS6S1TmezK8FFS30oJPuBCuF8kdQ/Y/IyxRGzpr7MGVndjNlUrLkrqWymBB1y45fz94ZhD6h6w+ZmuFNPN7nMFql/dNMxZKREX5LojmLNSIpLYsYFNcrQZM/uw3bATc1ZKOCs9KQGb4+gypWNl2N+N5/C48qQdTL+eHJPsww43eH3emjLgCJNFFnT+WD4Oq2emfVjfBWGOfciO+489md2NZ/pYMPamHMxKqIHV4aBWZHpKwDT7sNEdSNXH0omaw9cAcqIbz/xx/xnjzsSESkJe47pe3+qXZMw0+7DWEcbSKQMW7GXdBlan78xXzzb7sLQLwpZx/0lwrMyq1PqcYk1M/dDVy7bnlGtq3BIrOtCvWFk59Wsja/TzNlEQGzrUr1yLs0b1MvtIn/PjuhUukG2B2sjJaXqs4ya9bMZ8NdprP+wjTRiivf2KtTKrVhtzSrUgREsv2xvDxIZmzlGvAfVaM19aS8x83joCcez8CGjGHMXaNKBYy1R0iCU91ldsP2o3DCjWkvkKtMoVSENl2cGheh3TQhdfPQ==) **Figure : Build sequence** ### Set up workspace The Qualcomm Linux kernel supports the LTS RT kernel (6.6.x), which is maintained through the Yocto recipe in the `meta-qcom-realtime` layer in the `recipes-kernel/linux/linux-kernel-qcom-rt_6.6.bb` file. The Linux RT kernel recipe in Qualcomm Linux is referred as `linux-qcom-base-rt`. For more information on cloning the workspace and getting all Qualcomm Linux meta layers to use Qualcomm RT Linux, see [How to sync and build with real-time Linux?](https://docs.qualcomm.com/bundle/publicresource/topics/80-70017-254/how_to.html#how-to-sync-and-build-with-real-time-linux-) ### Enable RT kernel The Qualcomm Linux `meta-qcom-realtime` layer supports a `linux-qcom-base-rt` recipe that fetches and builds the Qualcomm Linux kernel for the supported machines by default. The `meta-qcom-realtime` layer applies the changes on top of the existing layer. During the kernel build, `meta-qcom-realtime` layer appends the upstream `PREEMPT_RT` patches based on the kernel version, and allows real-time configurations. Note - Use [linux-qcom-custom-rt_6.6.bb](https://github.com/quic-yocto/meta-qcom-realtime/blob/kirkstone/recipes-kernel/linux/linux-qcom-custom-rt_6.6.bb) for custom BSP. - Use [linux-qcom-base-rt_6.6.bb](https://github.com/quic-yocto/meta-qcom-realtime/blob/kirkstone/recipes-kernel/linux/linux-qcom-base-rt_6.6.bb) for base BSP. The following is the sample code for the RT base reference file: require recipes-kernel/linux/linux-qcom-base-rt_6.6.bb SECTION = "RT kernel" SUMMARY = "Linux Real time kernel for QCOM devices" DESCRIPTION = "Recipe to build real time Linux kernel" SRC_URI:append = "https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.6/older/patch-6.6.14-rt21.patch.gz;md5sum=91969a704a73aa918c89d3027bdd3634 \ file://qcom_rt.cfg \ file://0001-arch-Kconfig-Add-RT-kernel-support.patch \ S = "${WORKDIR}/kernel" KERNEL_CONFIG_FRAGMENTS:append = " ${WORKDIR}/qcom_rt.cfg" Copy to clipboard For more information on supported machines, see [Identify supported Qualcomm machines](https://docs.qualcomm.com/doc/80-70017-3/topic/getting_started_chapter2.html#supported-machines). ### Configure RT kernel - To configure the RT kernel, use the following procedure: SRC_URI:append = "https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.6/older/patch-6.6.14-rt21.patch.gz;md5sum=91969a704a73aa918c89d3027bdd3634 \ file://qcom_rt.cfg \ file://0001-arch-Kconfig-Add-RT-kernel-support.patch \ Copy to clipboard - To apply the external configurations on the RT kernel: - Maintain the configuration file in the `meta-qcom-realtime/recipes-kernel/linux/linux-qcom-base-rt/configs/qcom_rt.cfg` recipe. - Append the configuration file to `KERNEL_CONFIG_FRAGMENTS` in the `meta-qcom-realtime/recipes-kernel/linux/linux-qcom-base-rt_6.6.bb` file. - To add a configuration fragment to the RT kernel, make the following change: KERNEL_CONFIG_FRAGMENTS:append = " ${WORKDIR}/qcom_rt.cfg" Copy to clipboard - To modify the kernel command line, add the command line parameter into the `meta-qcom-realtime/conf/layer.conf` file. - The following example shows how to modify a command line: KERNEL_CMDLINE_EXTRA = "root=/dev/disk/by-partlabel/system rw rootwait console=ttyMSM0,115200n8 pcie_pme=nomsi earlycon idle=poll skew_tick=1 rcu_nocbs=1-3 rcu_nocb_poll nohz_full=1-3 irqaffinity=4-7 isolcpus=1-3" Copy to clipboard ### Kernel configurations Optional and mandatory kernel configurations are used in the RT kernel. To enable full preemption in the RT kernel, use `CONFIG_PREEMPT_RT`. The `CONFIG_PREEMPT_RT` flag is enabled by default as part of the `linux-qcom-base-rt_6.6.bb` recipe. The following example shows the kernel configuration: # CONFIG_PREEMPT_RT zcat proc/config.gz | grep CONFIG_PREEMPT # CONFIG_PREEMPT_NONE is not set # CONFIG_PREEMPT_VOLUNTARY is not set # CONFIG_PREEMPT is not set CONFIG_PREEMPT_RT=y Copy to clipboard To optimize the RT kernel response, use the following configurations: # CONFIG_NO_HZ zcat proc/config.gz | grep NO_HZ CONFIG_NO_HZ_COMMON=y # CONFIG_NO_HZ_IDLE is not set CONFIG_NO_HZ_FULL=y # CONFIG_NO_HZ is not set #CONFIG_CPUSETS zcat proc/config.gz | grep CPUSETS CONFIG_CPUSETS=y Copy to clipboard Set the following kernel configuration options when affining the RT task: - `CONFIG_NO_HZ_FULL` - When enabled, it configures the kernel to avoid sending scheduling-clock interrupts to CPUs with a single runnable task. - `CONFIG_CPUSETS` - Use the `CONFIG_CPUSETS` configuration option to enable `cpuset`, where the CPU is grouped to form a set. When affining the RT task to a set of CPUs, use the `CONFIG_CPUSETS` configuration option: ![../_images/compile-RT-kernel.png](data:image/png;base64,UklGRoIKAABXRUJQVlA4THYKAAAv8MMmAHcFN7Zt1c0WU8ghR2pAbStyA9gAc8jMF81uAW5s26qbLWbGmKkBNa1QpchMmSNH7Itwa9tWVS3cM8iI3HqmHSpwbobbd71y9vwnMFNUJCJSEdOcyMhEIE/FZBqTjCRREchREUhUJDNHRSBRBZElIoWCUZGMDBExS9KzzJKMLDcTmeVmwjPcjKiEZbgbd4QbioFLUbhRhRkixBURYoYMMUWmmCJTVCQTpkJCRFTkTe8i/4l/UvvQ6H7z+PdU8+w9etM7Ty/S8/jXbGU1Zxn1wjdqZ5miGkSYCCPivf39+XaXcBQLJfVKS6XUUCqWXH5mwdQCwNgcQB8nwM7jIFL/d/PxZlr54LvIpa7VjUqlnEqloF4vq9VK6vWyfxr1M57BE44A9EwAKxwtHXB4xmSJI3YOn8ETWK2wxMEzYIEn7AAeAXOssfUI7sAVm4AjlnjgHABwYGcGTvdt/l/XqnQMClJemOHLwMrZAWd1PnM2ajq/7IGU69/abwYWBQBS7LYZeQ0yvjzTM78z8zuHhDEFSmmY6T2nYebEjDJIrgVtJfXphZn/MXd7s7sz92733l6riP5TgiRJcht57gHAePQUBlONBo/ht8vXZ2uNODuqxFlfoo4XUzuY6bmlppld9ObVKLK1SpxKiTiLO2pslB563Rlc2hpFKgzVJz825jhhYJxtIXEGM5mVUqVXbadjYzvN5OY4iUC0d+aXibMhME7IZlugmf0hefoznZXSTa/nIJa+Gn12VemzvkSfxWlglwRsFom2zYIyeVYGFBl0OjZKzVXYJb1cQkGRMn02BOTZGZLnVq/DLimz2RdQJ6TPoHe0VmRPFpXJsypAcqsfYZfs9P4O+Hc9VnKbf1ejt9hONnoNJTupFCDssJM5/l0XvQEr2es3HLSSHbP++x9tG/y7tZaxzb/bWZwwN49a2eEgLOMOLvssYy2bckfWQS47c6n9/l04G72DxTA7HMFq/67kNazIsfo6FSqOYJeV9DiCrZ0KtRyr6jXM8++6Z/2XnVb6cJb7SUL+XVjot4nN0nxs0BXk0lMFvNlV9FrGfDbb8+m5QeZji9gEs/7LS+b5d912stkRLO6kqzqC7oKeWlHPevewseOt4h6qjuCAf9dT0LPMvysV09zk3y0qeukt9NvmfsZBykuXFM5NPX85vj05ihdUo4R7L0WGZyCRxpCQxvECqXizMoOyhnpGfh7SKL1ytFylbwuucCJhzgEaQ1SQYdrWeAOBDTBfYdn/DB+27d55D7KSanLuX9BSb7imqILyMH9QVSej3C6EYKmENp3SK0ab90FjKGOJkeYP0pZKRxX8PzeacOyiglxCfeem4qFdnY4PttJGjEPjUjS6f/4X6p8JkRxSmwhZYKWRhvJvgPqn0YgA4Ad16VEEpb+YrDxWenpLXPkK2e5g3nmz7z8T9e/1yEfOY9NKVyZBV/wguxQftiCKtCm/NyHeBJDSwoaUYdDSiES9WjJ16VEEpT+PVB4rPbkynRPHyc/+dRydDgtBNk3SWErh76zS3ZAl8IehVqPk6rzqvoO1nXYCcn4QeoZn6t+Kq1OgdbOmc1Py4zPqxuJ6M37QSQ/HyAvxQesFbEjZBa+aGGm+eDl6uxekKzqlxxiXHq88Vvr0OvvEfcd5ALQ4QWMoa5xNFa1GydZ51X0Hazvt58XLeuJnGHEtlOrRKDVNSvdANNrEnRAfViXyU6sX0CGl1Htn2CmqFvn9i4vJHUqn9Bjj0uOVR0ufWl0/Adx3nJa8c4w1699TMl5KyQBvvn46NfQaJV/nlfqOZudtu5tvwAt6RqVmc051843mS5eFJnmGIkYTxa9/r77P0CGl1NtPvH2GmaJqQkzKL5VKNEqPMi49Xnm09On1EUS57yQ1qVg/iOSsVzbahGOX8KpJi0n1z+MdUWMpxWQqjWyNyVQ6hU4AvH46JXQbJVfnlTZBt/NyWsZCTDaG6IhReP6ypvqnUQHb20mA6EKaiOvTUteIaZQepSx92zsBUn46SUwuDjLrjf3elGa5GOSRFdFYSjGZSmNLKSZT6RTe/AjEF1JB93HD1XmlTdB9IHIqsRGT8uFO48rwjIqQnilp7lcj8lPulvZ9hg6J3PouojwG0gVWUmnwXPsPtGIapUcZlV4NLT25vzwFWnn0Po08AJC8QCjZbdBZ7/AM/Hj53AS8gMIWk3SWUgym0uhSislUOoVeB3kfTC3KzptRbsSlmoTP4l1A8Sbyi2FKwzONIV3R2FvxzvOtEFejGquhQyJXoXIfJHYvxjdBSyNG4cfL8isn4/XvdUqPMSq9Glp6at8BS345Qg6b9UbkWuFzzlFILiZpLKUYTKUVSykmU2kqe+7586vvEuV+SO6DXJC+q9ko2TqvRHdD0oJ4ZXhUfu0P/UDJoSjFy9zahmcSo5VmURqwIdG682kgzxNvf8P2gZKlSTTAONFwxgF+0Cg9xqj0aljp6f/zDIcn/naIHNYEE8sB8TEWM56YKPxBvZRiNJU2emaFNl0qL7MU/41DPJC+q9souTqvRHdD2s29UgMYa8JL51Vf8HN9Go5dmFKK69/SJcbjXWtkGuD3T4UOZEjEfgWOPPlXLkhp5C/4STTN4Wl44bJG6REmpdeAlZ7ewaNHj6P58F39vAVIU2W4mZDGUorBVDplXgGWPHuIA9Z3NRslV+eVd0Pdzktuud2+Kemupzj8/bDF/9rXTySPv3uW5d1Gmi3VUfNcdKhIzqrxKxpLKSZTaaOlFHSWTXblT8/9dp9ZHtbw8FcsX+GH9l3NRsnUeRN11+y89Ep203vo6NfHifJQwuunqwct7j35Esfr/40hfNY7PAM/iPhNkFk1/iY6SykmU2mTpRT1LDtVpwr4puy+jzP1j/KwnzS+ySOJS5b/e54fiS+xfYUfNuuNvNaUlv2RWbXiA2kspZhMpbGtMZpKp899APIXPuVbZ5KX7O6kdEk2QKku/SwGbNYb3zzahLsvIbNq5At+vpx5QW8ubTCVRrbGaCqdPmefkL/OMe96H554O/qyLts7dTbVp/uLfItKWpMmH0hf1px/PRh/Ra/dUCXNRpr/uCzOTUIrA71zHDs3SCuab+0rXV1uC/2KiJxwfs8Wmzn1E76J48K/IqLT5aDrSWU8J0c43d/Oulx6ur/+3aL/42Nn0ctaNqv8u3IR3Qr/rmInK9zAQEddyQ30FPVUi3oW+nfds+KoFfTMy8/6Owa6HcE6G+q1jAUdH9X8ZVFHwa1sJwPqcfVyCYteyrP+s5st/l1gGZvY3ORvBWwW5WDr/LuqnSxzA7d01HW7gZoj2OjfVYp65vh3XbP+s5QDHXahC+jxusoOZ5mF9PsOJQs5yGLzH3Z7Xg7w+0zX+HdrZ6l3a5k6C+yEOreGtNkXEGdVmTZLOzDaIf0hefYH9FlZTmYDF/uEzc1hMgFx1paJ4/jsE6eTWekNiRMgYbO/rJf5+dKyEkEqVeJsrVEnI9gnXFKSwZA42wLabCiTZgvPCX2XGFJlSI0hRuyXxSWCLGbYFvJjo73YJ7FcEtseG9toJrfcFXiTzCnRZ3mVPptr9DnAwtOnCw==) **Figure: RT kernel verification** ### Build a RT kernel To build the RT kernel, use the following commands: MACHINE=-- DISTRO=qcom-wayland QCOM_SELECTED_BSP=base source setup-environment #verify realtime layer is present bitbake-layers show-layers | grep realtime meta-qcom-realtime /layers/meta-qcom-realtime 12 # compile bitbake qcom-console-image Copy to clipboard ### Test RT kernel A suite of tests is available in the Linux foundation RT test suite. The cyclic test determines the best-and worst-case latencies for an RT application. 1. Access the RT test suite source code at [rt-tests/rt-tests.git](https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/). 2. To include the RT test suites as a part of the image, change the `layer.conf` file in the standard Yocto build: IMAGE_INSTALL:append = "rt-tests numactl" Copy to clipboard 3. Run the following cyclic test: cgexec -g cpuset:core1-3 cyclictest -a 1-3 -t 3 -m -l 100000000 -i 1000 -p 99 -h 800 Copy to clipboard For more information, see [RT-Tests](https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/rt-tests). ### Debug RT kernel The following methods are used for troubleshooting issues in the RT kernel: - How to verify that the underlying kernel is real time? After the boot is complete, run the following commands to verify the kernel type: uname -r 6.6-rt15 uname -v SMP PREMPT_RT Copy to clipboard - How to make drivers RT compliant? Ensure that synchronization primitives do not break RT assumptions. The following are examples of the scenarios that can break RT assumptions and cause unexpected system behavior: /* Acquiring a preemptible lock in non preemptible context */ preempt_disable( ) …… spin_lock( ) /* Acquiring a preemptible lock in non preemptible context / raw_spin_lock( ) …… spin_lock( ) /* Acquiring a non preemptible lock in preemptible context / local_lock_irq( ) ….. raw_spin_lock( ) Copy to clipboard - Debug RT kernel configurations The following are the debug configurations for the RT kernel: - `CONFIG_DEBUG_ATOMIC_SLEEP` - Verify for sleep inside an atomic section. - `CONFIG_PROVE_RAW_LOCK_NESTING` - Allows the `raw_spinlock` vs. spinlock nesting. Ensures that the lock nesting rules for `PREEMPT_RT` kernels are not violated. While debugging, you can switch from an RT kernel to a non-RT kernel. To switch, make changes to the `meta-qcom-realtime/conf/layer.conf` file to change virtual/kernel settings in `linux-qcom-base-rt` and compile again. To make the changes, run the following commands: - PREFERRED_PROVIDER_virtual/kernel = "linux-qcom-base" + PREFERRED_PROVIDER_virtual/kernel = "linux-qcom-base-rt" Copy to clipboard For more information on locking primitives with the RT kernel, see [Lock types and their rules—The Linux Kernel documentation](https://www.kernel.org/doc/html/latest/locking/locktypes.html). ### Tune RT kernel Tune the RT kernel to achieve a deterministic latency for RT tasks in the device. Set the CPU cores that run RT tasks to run at maximum operating frequency while preventing thermal mitigation of CPU frequency. For example, in an idle sleep scenario, RT tasks face scheduling latency due to CPU wake time delay. For a system with eight CPU cores, use the following system configuration: - RT tasks as outlined in the following example are affined to cores 1-3. - CPU cores 1-3 run in isolated mode with all read-copy-update (RCU) affined to cores 4-7. - IRQs are affined to cores 4-7 in the kernel command line arguments. - System sleep is disabled for CPU cores 1-3 and is set to run at maximum frequency. # Disable low power mode and sleep for CPU 1-3 echo 1 > /sys/devices/system/cpu/cpu1/cpuidle/state2/disable echo 1 > /sys/devices/system/cpu/cpu1/cpuidle/state1/disable echo 1 > /sys/devices/system/cpu/cpu1/cpuidle/state0/disable echo 1 > /sys/devices/system/cpu/cpu2/cpuidle/state0/disable echo 1 > /sys/devices/system/cpu/cpu2/cpuidle/state1/disable echo 1 > /sys/devices/system/cpu/cpu2/cpuidle/state2/disable echo 1 > /sys/devices/system/cpu/cpu3/cpuidle/state2/disable echo 1 > /sys/devices/system/cpu/cpu3/cpuidle/state1/disable echo 1 > /sys/devices/system/cpu/cpu3/cpuidle/state0/disable # Enable max freq for cpu group 0-3 echo 1958400 > /sys/devices/system/cpu/cpufreq/policy0/scaling_min_freq echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor # form cgroup for core 0-3 mkdir /sys/fs/cgroup/cpuset/core0/ mkdir /sys/fs/cgroup/cpuset/core1/ mkdir /sys/fs/cgroup/cpuset/core2/ mkdir /sys/fs/cgroup/cpuset/core3/ echo 0 > /sys/fs/cgroup/cpuset/core0/cpuset.mems echo 0 > /sys/fs/cgroup/cpuset/core1/cpuset.mems echo 0 > /sys/fs/cgroup/cpuset/core2/cpuset.mems echo 0 > /sys/fs/cgroup/cpuset/core3/cpuset.mems echo 0 > /sys/fs/cgroup/cpuset/core0/cpuset.cpus echo 1 > /sys/fs/cgroup/cpuset/core1/cpuset.cpus echo 2 > /sys/fs/cgroup/cpuset/core2/cpuset.cpus echo 3 > /sys/fs/cgroup/cpuset/core3/cpuset.cpus mkdir /sys/fs/cgroup/cpuset/core1-3/ echo 0 > /sys/fs/cgroup/cpuset/core1-3/cpuset.mems echo 1-3 > /sys/fs/cgroup/cpuset/core1-3/cpuset.cpus Copy to clipboard The following example shows how to add a kernel command line parameter to disable RCU callbacks (**rcu\_nocbs**) in `meta-qcom-realtime/conf/layer.conf`: - CPU cores 1-3 - IRQ affine to core 4-7 - Isolate CPU 1-3 KERNEL_CMDLINE_EXTRA = "root=/dev/disk/by-partlabel/system rw rootwait console=ttyMSM0,115200n8 pcie_pme=nomsi earlycon skew_tick=1 **rcu\_nocbs**=1-3 rcu_nocb_poll nohz_full=1-3 irqaffinity=4-7 isolcpus=1-3" Copy to clipboard Last Published: Dec 29, 2024 [Previous Topic Getting started](https://docs.qualcomm.com/bundle/publicresource/80-70017-3/topics/getting_started_chapter2.md) [Next Topic Yocto support](https://docs.qualcomm.com/bundle/publicresource/80-70017-3/topics/yocto-kernel-support.md)