NVIDIA/Mellanox ConnectX driver (mlx5_ib on-demand paging MR creation): The xarray tracking implicit on-demand-paging

Impact

The xarray tracking implicit on-demand-paging children was not initialised when an ODP memory region was created, so deregistration walked an uninitialised structure and faulted on a wild address (the report shows a fault at 0x800000000 from ib_write_bw). ODP is the mechanism that lets the NIC page tenant memory in and out on demand; corrupt bookkeeping in that layer is bookkeeping about which pages belong to which memory key.

Who can reach it

Local, unprivileged. A tenant registers an ODP memory region and deregisters it - the ordinary lifecycle of any GPUDirect workload using on-demand paging.

What to do

Kernel update initialising the ODP xarray in reg_create(). Disabling ODP is possible on some stacks but costs the pinning-free memory model that large-model training relies on, so patching is the realistic route.

References