NVIDIA GPU firmware microcontroller (Falcon): MULTI-TENANT ISOLATION: unscrubbed microcontroller memory leaks data

Impact

MULTI-TENANT ISOLATION: unscrubbed microcontroller memory leaks data to a privileged user. Same residue problem as the unscrubbed-registers issue but with a bigger surface, and it is exactly the failure mode that makes bare-metal GPU handoff between tenants risky: memory that should have been wiped at teardown is still readable by whoever gets the card next.

Who can reach it

A user with elevated privileges on the GPU host, typically the next tenant to be scheduled onto the card.

What to do

NVIDIA shipped the fix in GPU firmware/microcode delivered with the R470 and R450 driver branches and, on some SKUs, in an updated VBIOS. On most datacenter parts the microcontroller image is loaded by the driver at GPU init, so a driver upgrade plus a node reboot applies it; check the bulletin's product table, because a subset of boards also needs an out-of-band VBIOS/InfoROM update, which is an offline per-node flash with the GPU idle. Either way the node has to be drained. On any fleet that reassigns bare-metal GPU nodes between tenants, pair the driver/firmware update with an explicit GPU reset and memory-scrub step in the reprovisioning pipeline; do not rely on the card clearing itself.

References