NVIDIA DCGM (nv-hostengine, DIAG module): Any local user can inject a shared library into the DCGM server

Impact

Any local user can inject a shared library into the DCGM server, which normally runs as root - so DCGM turns every unprivileged account on a GPU node into root. This is the single most operationally relevant NVIDIA bug of 2021 for a cluster operator, because DCGM is not optional infrastructure: it is what feeds dcgm-exporter, Prometheus, node health checks and scheduler telemetry, so it is installed and running as root on essentially every GPU node in a modern fleet. Combined with a container that has host access to the DCGM socket, it is a container-to-host root escape. Versions before 2.2.9.

Who can reach it

Any local user on a node running nv-hostengine, including anything running inside a container that can reach the DCGM socket or port - which is the normal configuration for dcgm-exporter deployments.

What to do

Upgrade DCGM to 2.2.9 or later on every node running nv-hostengine, then restart the service (systemctl restart nvidia-dcgm) and restart dcgm-exporter containers so they link the new library. No GPU drain, no reboot, no firmware flash. Until you patch, stop running nv-hostengine as root or block local access to its socket/port - the DIAG path is reachable by any local user.

References