# Feature overview

These are the currently supported features of AICM. For more detailed
information on specific features, it’s advised to look at the Swagger
UI interactive docs. When operational, AICM provides a Swagger UI
interface accessible at `<IP>:<PORT>/docs`.

## Metrics retrieval

AICM provides a data retrieval mechanism for the following metrics:

- General health data (same metrics obtainable from the `qaic-util`
tool)
- PCIe data (as in bytes read and written)
- DDR data (as in bytes read and written at the NSP level)
- RAS errors (from various components of the card)

These metrics are continuously requested from the cards, and their
interval can be set using command line or the config file.

## Groups and device configuration

![](data:image/png;base64,UklGRnoYAABXRUJQVlA4TG0YAAAvxAQrAPXgYbZ/ktw4GzJ0qFChQoYMO/OEHTJkuPcOHClUSGcMFTL0bYcK994NGfo2Q992qMzY+n+/f1VX/buOVns5wmDAwl4s1Z4kdHX5FA2aOgYNDH8soDA6fhxswffR+PtUYcSa8iGqfElbPaCh3NoeOXldI9nBlZ8QObKysjISOTJyZOTISmQksnIkO8gd4njlx/7JV0K5trdu80JBQ0NBQUHDtrvC3n7c7jMMDCwMDCwMNDQ01NlI/yVBkiS3TQ1JGxfZA2yvFjwXeP4OmEXFr4PvB+8Dmsh/AQSl/QfBYvEpYOd94uPg/eAkUGyAy1Vwyh8Fi8UFYH+XStxYqKjmu/ghsHgQQOt+Uc13XS/Pg01dr/U7tGu7gMp0rZ5yVVxeXARQw95WEpyhxeLaPBLV4ixwAe6mgmmLhTvry9bemEu6NIvbLi8tXMZWIalX3QBR1xaX55GAJh8seE1htfXjeu0GLr9ezyehEUCpSKvwhkduzCNxe9HMg2EXa080OHDD48CNeSSaReOCfpbbdDd3TWYhL6POI8VlfCPaGmfRvfH2mqOHLHdjgSwapW9u6X22aum1tRrsNUcJaelVSkFFBm4ofXNdoRl9aUnmw+LShPzyFueXnE1s9pwujNYYFAfvLI9vzt2MObdi+33npOn3bXiocuoUg6J8N03T7bfnakY8csKJifRkZTPzWBcX5/zyIYbF/uGVb6TV/bkZjh/YkI5HPh5R1d2v7v8ZFiqTj0eeSv/seD6Gh3vS7Z9agNN+df/fsMBJX9pOZQ6Gx2ccpO8fGAZnmqGh/vP+HOlBN/fCQXo2xXDH8BhW6cH8y9TVYHyoizyhyechj3oK/Ti9TOkoc3YrGR+PbYlzef8Q6fETmSzGZPOhhXy8n+3EGQ+W8X7fjI8N8dTgTNz+s8TZaIhDcSVEufVT528hH6cXa/rw0e08q/FxF/hSej9xNhriT9xdhCo/dv0V5tGfb/XEFKdy2zv/UU0P4O/vDOOR6may5FYff+JWyPLjVx/hHXl/Jq7pI+E2PXw8kr6dNMnjToQ3uVHhpaYzKyvKyaE9uB57b00JzuVkZTtxYrYlmI13s/krLbzJe6Mi99BaMzArZzgdr+4a9owKj2xm0ptJlbwPUcPlzKb7OfzJ+/P309n75ZYxcftv6XbWJ2VWVsDm8ZI+sjVT9s7Z/Igtd42JN5/e88Zhepw0MdoSzGywm033c9iT/9B9PcNxynSDOqKNWUyJ/z68/QpuZ5ekzMpKcaZb00dtIuyd8X5KeQQdScfuhsTYbe+FJ8+zemI/XSZT8lfOtHE4MhvRzbHEyOzwGWI4fVZGJmXwSgt38ldGTBv0zYejUXF4HbezLinzsnLYSFtDryPJhuz9/mpU3Pnanivq8u47SROdJULUcIcjZlU1m+7nMCfvz7QYUzcfjpsvV6OCD74FOwq6pAzMKu+PHqgD2Tut2JC2H/55NCraP+c9L5TjQtJEZ4kQNRyPaZCU6X4OaXCX/pTp5sNRfuDpbFQIH8KevS4pE7PqT5b20NiBvadf+ML1Qz9PRoW0Lzy954HvbidNdJYw3HBmJmW6n0Ocs/fLf5vHf99W6JIyLystb46g5rH3ZhJGZ4kQNZzwLiRltp9DnVtjFvMY0eHpDl1S5mWl5e6WeYykd5MwOksYbjgzkzLbz+HOmDvz2NVyw0B2XRi7m8f3EjE6S4So4YR339+6wmg/z18jIkYjImYjIrMGwsNwImI4IhKtCMYkK5lZAkT8ySpQBGOSqrFtZWaBGG44f5LyiUD9HKUIY4h4XscXpDZjCQxhmpV3yb8Nsee4YwkQaTE8/0fa87Dhp7eklVkAwVnCL8MF4+afMre/rMQSIMKT53VRyX7cjSU6EbYQBbMYiHBpIUoWM5H2p7dkgGhbmXEwqSUMwkW0iopZjES4sRC1k1dcoGAnaCmijWNxpKEMZq0gKsBPSmZVTlmvmWZAsiJ6NNn5TOvEergQpBXNTi0+A4cy2lYniKmAJI5BlhAfqdiJymb+utlX1Lxq0Y7Fy0rznlfEoVZ2AdtBSMteyLpnxjKkE0QE+IqobV1pNWtTEW1SurTFX1QV+fGtPBy1QndqSZWpZOgUuAhI4BhkCRY/EaENMZqUNze7CPAZJw3SSvWy0hBxry0iDrU54FE4g8a3sCpSIAoLGbbLMxamAnxlqdIplWbcb0AVSkzHhVUKOiWoYbJvwSz+oh7g4hbqUdr9wYmzKm27duYTEHVv2FEKdhw9a62A5I2fliiVp0CUmyV0AvxkuUT/auzElWUBUbT2q61FvxfWKiW+Ahl0evlYd4Ao5WnN2tQKiDiWAhuWC7KJuSZ1WQmQnV3bQfsZN5nPdCg3g9JJS05VsQNJj8sKs+VCFVHf2Rn22/bImQ8U/FTQ4ZutqGkA2dldH7XtatW27bBqJXHjpyWY1mWuloDaiGu12+gzhW10diJVWofWozYvnGw1VZ+faCpnss9siYtFZQerCSuJxja61SpRB2zhkm5cGK+0qHVekrpU1JauWARl+AjJjiu3ctnYjGALTKqzJeip/Qeqy3tqIWmHdavGKy37ANalaqkj1TJEGUkb/yyhaTXY2tUSDbYMbeE7me10diJVZmYbqNfozpWTrKbSE5/BVZW5NnSKipHCQlq61RqNQGXJDEXHFg5VQahQHJYqpaULAKwscVNntgPxAHcgTqBU4aYOBqgsVUtvpQYddoYW2UFxApKUlqPkja+WKG0JTnCzBJgE2g5BoLWTIwcrN6S26GaQBEkGA+ThZKJpV9qagPu6ol+tEh3sTkCBGmHDQuu3JNBt1ZAsJmF3AjqiGf/+YUEEStvR1DGC4mjAWLUgox12COsV3nm7QmVJHF8tAQ3C2rpbIiPCJmHXO1o7KRlQlemch97HmIzdSaiI6ozputIkCOI0qzUKqfBvCYbAtMyyuJNRZUvPTPCER2dLmhUkaDMvaN74GMjoo+EeXVBN2x/2PbBuiTJO2vhpCSz8Ve2FhlrCZwrb6OyEvi1t44Fa42bfsZVDQfOC9VRXLO6Umtoi0hCyNS0LhNr8NVe1hoJSU2V+onlKIiOalZ6mtA1CxSkaVLYMYLEVzgnvP1OV6UoNCYEOrFYoLIEv5q8lOts5Q+6WyGxHLOE3tmCtnZxkCjVEbU4HbEHd7C9QLWYOla62cHTYTEMNaFfrMtIQ+FMFZQVEMD4xZjtKoZnFojB/EW7wmb0KNeNGrp18kUY3S0ayCuD9ad9syds4RN2osasjyhHOQqvXRH7jsyWqOqs8WKKxta+G0yRS205rJ2w12JKSYeq61+v8RuCtDLRiZ8irrjrbELgilYRmtUYbavXiBqigtW472LxFxUJoQF3VcQcCyorFP6BSxAygtZ4VsJELWxIEChF3FTsqOpVeyeLzJwV28IF21KLtGjSs1sM9grSrQYk7atthgFnuJe/RPlsiq6rMiyVqkFlkLP7BzGgurZ2UzS0Lgk8dcZ05KipYByy+oioIrHStkt9ApdlZ1cAkdKDONtrVGlloXkIQpaKmrxaoLVtT1NayFtrHoLPy8Ub78gMRX7CgoIZCnq5tmAhwrn5/ZnPVkrvHVwtgcFi1QsBHW7eCOuGatPHZEp0jxoMliM7MxxuLiaistHZS2RQaQB60QFGnvzcWVgPJq6CvFghbm1HIo5WsXa0RxVLoU42iVSFEgpB70ukRIcK8MFFS7CEpIsDzZ0qC+vYM0b1aQPZVvX7cqU3gow2xxASGC9DNS3F3s18shf3Jy9NKiyyW5BuRXFRQCbp7Ej8+wjpRUuIhKV0WLoIEsgrue8y0InQfNNcI0ipL3hhiiQkMF5ybffrguWc/+5KXp5UWNZj87cC78fJrsauPS35IaTFN8Z/vVRH4AyUhZzhvmOXnBMGWeWyFGBcgZ1xPmm6Yj+kj+phlioK3T5qecb2bKLhlPjOx37s85/mk25lWNh+O0xenufvDVxP2713OvH55/KchW2Pz97fjFzbh9AVv261y5vnL46Q7CWOaob1V0CVlYFYKo/uAEU6cdt2gq1expi+E+fCfZ559wCAde/m9DRazdn3+ifQbRpeUgVnpeGtvpGRab3wu770VTrw6Nne8spzGEOGx2auzzt746LpYNTI354nqG5ruYbkkZVhWet7aO61hUL/INuR7L7x6nvWtcOIC5A+dqcvpDGfiH95Nyv0ii9k9VKSY2kPFkOKM6+m2U9epvRzuJuQeKsbsvsJqMLSvsCFFddJUXaf48rUqWfcVNh732lqHmb22Dive2zCfcrbfm2322jqUu88f0fznmD7ljOn/GT3EkZiFsRizTDkj+mIujN1zypf854vxkuf0AbzrymfUaN4oPonhEhR53+f0ATzr6vOZLxP4ifrKuyETE3n/uGJL5Z339rc1+tybrou821f7fAZN3l/gan/ZWp9P4qu8f9H2eVJD84FMlEDJZ9zoDUcUEPJkRq75CD9KyAE1cbaNi+FyFNArnIlJjLz/+MNsrSWlylq71fcWQvp8xk3eXwd7UQWvOnJUvNjnCYo8R/rr9nrfP+NIAB39q7/tvY+lNyKSHMgJ/ZYzrb+qLHbdPtP3W5LAblRAZSn2Kv4NfKZ/BspUHxajzwUJQfw/mo2kgIpXaY2Zowdf3FIK+sQ12pJ4EQqSg7o7mF33F5LjDx77AeFVF3QgRxIBJBzwFVBsNbyobvKcLpbEbq7Dk25579BfhSnxCS89sqmZZSKSxs3Wi+R5uWfg2ehXex3JarQ7ufqzaF/t4xk1s1q0A5Fkl1bXMRI64NaZkfg+2gO9ahtu9cns/WmUZ2CAlKt+y8YnAjmaDdduJcX1xShbwHVc2NFyvU9muM0i+N60F2McS+EOH6WAXVqqorMZ6eAbu3TwrWOJ67jN8iK+N+0ZSp7EyNWL2+Tv39Wrqi6l8+UxjKUwHj9PQTpDWlWK2jbQ1Urc2W2gq5Ul6Z1qnCfvre37/kWn0SD4Hg+B93gktjfZvmqdeAbnsPYqldLnsYzCao+Fg7T7qoKRAgQ5Qfp1WcV6qM22cA5r8a5fBOMlsG/P0H/qw0VAznEN7bFwdHiMCRkB93U1faivLUtc//YMF5vl7s7Lk9j3mOk/f+sigOMblkQJg5zZht0OBKEkyuI3TN3kwWYuhkuY3ygb10ZDQ4+86gnTioLFnY4smFC+UXbWj+B+qy2YdnirsaUzpKGmlKhsOQ+E56PZ0B4LhzhjS8vidjQbtHqdE7rxfDQb2mPhUI29ovZyNBuZzdQsNcscEJ6PZkN3LBwoqHQ/mg0qoH5C7CbGfXuGiPZgNiY9mo2ZAfG6X2Qx6nvMdBpkwqPZiHjyUDWc+Wkl1x4q/kXi/VrsvA9Xw5mfVpLsK2xcIWb1FdbcrGJLX2HN7V+y6b1LdmPvSYN6bT1223vhyfOsMYWY1Wtrg7OKK7221vT03fRuxJuXlZ69kcS0/vN/bS+eELP6z29yVjGl//yaY64I0QO6ED6YgiMZiSXErCMZMTor849kJFEeU1ICRmeJ8DWc4ceUlCyP7i0Jo7NE+Bouvhzd261d8xjRb+laxLsG8jXTj7NyJL2bhNFZImwN52B2WuHOmDvz2NVyI4SPeDcRo7NEyBpOYXRa89eIiNGIiNmIyKyBUDGc4YhItOJPTyICRALKKoCzQDBJ1di2EstYRCIy/YYTnwjUz1GKaL6DwfM6zmzGEhjCQWUVxPm48FwznoepcwniK4n1mJImtITnta9+DNkIw3mXbC2LPwTm59qWLNGJsNWcwxFz8NpPCSORdj1AtK2YQmI9ujdvligtRMliEC5+guhYAscHP8cFCnaCliLd98WgLBFmrSAqYEJ87KeE96z8Rf27HnZADd2n1ZxBEBjwcFJBJiI+HGdlRFIycwmV2ASW8HBePvykc3RntvKelTe/ecMsP0ccamUX5Mz0MKx2um2ZsQzpBBEBk+FjPyUmySqAswV11ArdqSVVppLh9fTOyCTEhyPejUxEuIH6TTxaQpZevsvST1QWnS0mz0ovQLwQrJ/FvbaIOGBzFCzY/K4Yf5IfRNFCVpETbUQF+Ajo73Bj17DDjDmCKA9ZBXCijbBOXKlHaffxNBq1cKaglCh6oo12hmFYawVMQGw49vAIBaq5UlfgHTUZC4oqLFZ/TjSsE+AjkFWt74cH5oiiqgrzc0InwFNWQfnZpf8fWj9HHEvyYz2FZVU9MvxqD5ah0pZMdnbJT4/hJvMTyK5ich6obMWaE7HUuWbV2YyUX9/ZGfbxtIyBgm8O32wFpt0bdlqys7s+atvVqm3bYdVK+GHPeR6bOe2B6bwKj2mv2shEU//oCjzsEKH/Ckucx2XFsF+KVaG/qFx0WWHDDfwHYDJZgTbnQtXN7gTlZ8gIK4nGNlo/Rx2w/UrykxYZjFdaGIqP5pfIuNB0k0R8BUtj5lohF5V7VhUpuf4D1eU9tZC0w7pV45WW8zAA61K11JFqGaKM8MP+88jK5sORp/EizBv006Y2SkHjaQq8ptVga4It0BENtgxt4TdYf1UuFbLKAdDsjELfFsCkXgjKzwpal2r9HJHUlp2AomMLh6ogVCgOS5XS0gVCUTEWQ5CjcnDJipQqUhgCACpL1dJbqUGHnaFFdlCcgCSl5SgU+WnI1ptTemZU3twqIxhtgUcjdLYEuMNmooDToO0QBPqsMtuBa5ESxYmShHWcFwLyMwHEiZufo4PdCSgshAPI6GxJoNuqIVlMwu4kVERzhgWztJ1rVh3JKiiOBoxVCzLaYYewXuGdtytUNhHJ9hz13Nr1jr7Asy2wuy8abwhWhhiTsOsdfVaNrbEGpj31oJ7H8ERAfkYam7n5OQqp8G8JhsC0zLK4k1FlS5+xFXNV0KxgA9cVizslZhXQ6KPhHl1QTdsf9j2wboky9kzSPVtQ3kfrCzzsBVW1FxrqCJ8pbabPCpxtMy+w96yC8jPFi58jDSF/myyTt+GIKkBVraGg1FSZn8DmyxwqXVaODpvpsqJkmFUQi61wTnj/mapMV2pICHRgtUJhkywWB87bWAST2U5b4FXh75whhIrDyhAd4Te1bVyyKm3jDFFwANR5zyoYP2uoPfg50oBnGxsoKyCC8fkB21EK+pR/YVGYvwi85EoLIENWddXZxkNWVUWyCuD9ad9syds4RN2osasjyhHOQqtXv26iOqIX5YyCRVvgHQ11VrEgjW6WmhjOV6B56ZqVzWonFQBVCSsBmfesAvKztv8fLn6ONtTqxQ1QQWsdKs3GFhULoQF1VccdCCgrFh9RiWDtaC0Uw4ZF6S+8ZFWCzKxg8fmTAjv4QDtq0XYNGlbr4R5B2tWgxB217TCoWZSiGQFQbSlzaQu8cFZVGUE4U+q6irkGmUXG4iPspFO4Z1UXtiQ4IpS6snDyAZlV6SWrgPxM6ECdbbR+ji4gatY9tUla4JWtKWprWQvtY9BZ+XhDnuokWRXkWU1IN/OUVUOeVvX7M5urltw9vloAg8OqFQI+2roV1AnX2QCWemypLfCq/rEdhTxaw0Rn5ueNBY+5ZgXyGwraXJde6ekmED9ThDvIRu/niGIp9KlG0aoQIkHIPen0iBBh/rAU9j0rxqyC+vYM0b1aQPZVvX3cCYYTN5NZwoMjiAA/P+4UdFZu6XnAND9HFktPPYkQKkF3T+LHR1g9JuV7VgJZBfc9ZloRug+aawRplSVvJrOEF0doLLFc+vTB8+CzWk6WlWF+jhpM/nbg3VD8WuxET8gazrwfKEkQbJnH1iyfZPl7l7GE5Px7l0nyl8djCcn5l8cTZR8w4giJuQ8YibI3PrGEpNwbn0TZL7J4QkLuF1kM76HiqdweqGO6eqj4UOW/Ip9k3EPFON5X2I20nZhiuvoK++47kU8i7itsLO+19d+lj9q0NzW9tr4zXJyTnk1EPrPNXluHeRff/zT9k2nisS7S48hnttl//hCP/lr6/9PElfTmE5mEe5wc2l/tmR8avrFtkyYxJ0KWMbkV7jHqejo+6u/3zfzYEIo7w8cj6dtJk9gTocrnIYuQj7MhnZg4WeQvpiCeB8Yj1c1kyZjcij9xd2Fjc9xPvzSYH3D9UnrfJkquLOJQXLHxOf5se7hjfMD07T+z8yxxnL5/iOHO3xsfd4bvpcdzLdiHe9KzKYY75l9Xqdg5lzhdUBej/3NHXQ/svAuPzzhI359jcP5j8P+cqQcdMO8ycftLAzyYedPU/760nYqdi4njP0uf+ngE3oRhWKhMPh55Kv2zYzsvE/er9BubmcN9096Xtn945Rtpdd/O0cTNt7fTNH23/K5BUb6bpun22zftnE3cPF6+c7BtUBy8szy+aefD4pL/XPKfS/7zxcmwAA==)

AICM can work with individual devices by querying their id or using
groups. Any metric retrieval API has a group counterpart, and other
features like policies/policy-alerts can be issued for groups. Groups
are identified by a unique id that can’t start with numeric characters.

Group management is done through the various groups API. It’s possible
to create, modify, and delete groups and their members.

Groups are intended to be only logical and limited to a single host
(can’t span over hosts). There is no check on actual devices. Once an
API is requested for a group, the logical group request will result in a
data response, and the dedicated field will be
`device_reachable=true`, while the unavailable devices will return an
empty response apart from a single field `device_reachable=false`.

Another useful feature of groups is the ability to create and assign
device configurations uniformly across all devices within a group. This
ensures consistency and simplifies management.

The user can create a configuration and assign it to a group, this will
be the **target configuration** for the group.

When a device is added to a group, the target configuration is
automatically applied, but the enforcement may not always succeed due to
various reasons, such as the device can’t be reached by AICM or the
configuration being incompatible with the device.

In these cases, for some devices there is both a `target_config`, the configuration that you wants to apply to all
the devices in the group, and a `current_config`, the configuration
that’s currently applied to the device.

There is a discrepancy between the two since the enforcement of the
target configuration wasn’t successful.

To see the difference between the target and the current configuration
AICM offers an API that generates a report where issues are highlighted.

Example of a report for an AI100 card with target configuration asking
to have LLM mode enabled. The report shows that the target configuration
is different from the current configuration because the AI100 doesn’t
support LLM mode, which is currently only supported on the AI100 Ultra
and Ultra Plus. The user is notified of this or other possible errors
through the details field.

{
      "group_id": "ExampleGroup",
      "dev_config_reports": {
        "0": {
          "device_id": 0,
          "target_config": {
            "ecc_ddr": true,
            "ecc_vctm": false,
            "llm_mode": true
          },
          "current_config": {
            "ecc_ddr": true,
            "ecc_vctm": false,
            "llm_mode": false
          },
          "details": [
            "Failed to enforce LLM config: LLMOptions(llm_mode=True) on device: 0 - Cannot enforce the LLM Mode. LLM Mode available only on Ultra/Ultra +"
          ]
        }
      }
    }
    Copy to clipboard

## Policies and alerts

AICM will continuously query data from the available qmonitor server in
the background. Users can set up alerts for specific conditions.
Currently, AICM can monitor these conditions, with plans to expand these
options in the future. Available conditions:

- temperature\_violation
- power\_violation
- DDR\_pages\_retirement\_violation

In turn, it’s possible to activate some actions.

As of now, the available actions:

- Alert action

These can be linked together using the Policies APIs.

### Alerts

With policies using the alert actions, the user can create an alert
mechanism.

The final user will be able to receive alerts by subscribing to the
/policy-alerts endpoint and listening for [server-sent
events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events).

This is possible thanks to server-sent events, so the user will need to
parse them accordingly.

A complete example is available in the `examples/policy_alert_subscribe_example.py`.

## Checks and diagnostic report alerts

AICM supports the ability to serve diagnostics information from the
cards leveraging the Field Diagnostics Tool. This will allow the user to
detect and troubleshoot common problems affecting the cards by assessing
the state of the card before/after inference runs.

This is possible by creating a check on an individual or group of
devices along with specifying an option to issue either of
quick/medium/long suite of tests.

- Quick:
    - Involves default software checks configured in qaic-fdt, PCI checks, device checks, and quick inference test.

- Medium:
    - Same as Quick plus ECC checks and medium inference test

- Long:
    - Same as Medium plus long inference test.

### Diagnostic report alerts

The user can also subscribe to the diagnostic report alert received
against checks created by using the /diagnostics-report endpoint and
listening for [server-sent
events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events).
A complete example is available in the
`examples/diag_report_subscribe_example.py`.

Note: Special cases of errors reported in the diagnostic report alert:
1. If there is a problem with a card, the diagnostics alert will report
error “User permission error” and prompt the user to run AICM as root to
have complete device access. 2. If the diagnostic alert reports an error
“Command not found: /opt/qti-aic/tools/qaic-version-util”, the user
needs to grant appropriate file permissions (chmod 755) to
qaic-version-util.

Last Published: Aug 25, 2026

[Previous Topic
Metrics](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/metrics.md) [Next Topic
AICMI (AICM CLI)](https://docs.qualcomm.com/bundle/publicresource/80-99100-3/topics/cli.md)