NVIDIA GPU Display Driver, GPU hardware performance counters: GPU performance counters are readable by any local user
Impact
GPU performance counters are readable by any local user and leak enough about another process's GPU activity to reconstruct data it is processing - published work recovered neural-network structure and input images this way. On a shared GPU node, one tenant's job can profile a co-tenant's inference or training work through the counters. NVIDIA's answer was not a code fix but an access-control change, and the restriction was not on by default for a long time in several driver branches, so a large installed base ran exposed for years. Treat any node where profiling is unrestricted as offering no side-channel isolation between GPU contexts.
Who can reach it
Any local user or container that can open the GPU device and run a profiling-capable API (CUPTI, nvprof, Nsight) against the same physical GPU as the victim.
What to do
Update to a driver branch that ships the profiling restriction, then verify it is actually enforced - this is the step operators skip. On Linux check the nvidia module parameter NVreg_RestrictProfilingToAdminUsers and set it to 1 in /etc/modprobe.d, then reload the module (node drain) or reboot; on Windows apply the driver update and confirm the developer-mode registry setting is not re-enabling profiling. On a multi-tenant node also stop mapping profiling-capable device nodes into tenant containers. Note this is a mitigation, not a fix: MPS and time-sliced sharing still leave the counters shared at the hardware level, so the only hard isolation is one tenant per physical GPU or MIG.
References
Related entries
- NVIDIA Windows GPU Display Driver (nvlddmkm.sys): Resource leak in the escape handlerCVE-2019-5671 · NVIDIA Windows GPU Display Driver (nvlddmkm.sys)Medium
- NVIDIA Windows GPU Display Driver (nvlddmkm.sys): Out-of-bounds read in the DeviceIoControl handlerCVE-2019-5677 · NVIDIA Windows GPU Display Driver (nvlddmkm.sys)Medium
- NVIDIA Windows GPU Display Driver (nvlddmkm.sys): The escape handler relies on invariants that are not guaranteed, soCVE-2019-5686 · NVIDIA Windows GPU Display Driver (nvlddmkm.sys)Medium
- NVIDIA Windows GPU Display Driver (nvlddmkm.sys): Uninitialised pointer use in nvlddmkm.sys. Local denial of serviceCVE-2019-5693 · NVIDIA Windows GPU Display Driver (nvlddmkm.sys)Medium
- NVIDIA vGPU Manager (vGPU plugin): A guest VM hands the vGPU Manager a wrongly sized buffer and drives the host into anCVE-2019-5696 · NVIDIA vGPU Manager (vGPU plugin)Medium
- NVIDIA vGPU Manager (vGPU plugin): Unvalidated guest-supplied index in the vGPU plugin. A tenant VM crashes the hostCVE-2020-5959 · NVIDIA vGPU Manager (vGPU plugin)Medium
This entry is curated: imported from vendor advisories with machine assistance, not yet individually verified. Confirm against your vendor's advisory before acting, and report anything wrong.