InfiniBand / RoCE memory protection - memory region rkey/lkey namespace and protection domains: TENANT ISOLATION

Impact

TENANT ISOLATION: The only thing standing between a remote peer and a registered memory region is the 32-bit rkey, and ReDMArk found that real RNIC firmware generates rkeys in a small, largely sequential, and therefore guessable space. Combined with the connection-injection weakness, an attacker can issue RDMA READ or WRITE against another tenant's memory region without ever having been granted access to it. Applications make this dramatically worse in practice by registering one huge region with IBV_ACCESS_REMOTE_WRITE covering far more than the buffers actually meant to be shared - a common shortcut in RDMA key-value stores, parameter servers, and disaggregated-memory layers. On a GPU node the registered region frequently includes GPU memory reachable through GPUDirect, so a guessed rkey reads model state directly out of HBM.

Who can reach it

From any node able to establish or inject into a queue pair on the target RNIC, the attacker sweeps rkey values in RDMA READ requests and watches for a completion instead of an error. Because the RNIC services the read entirely in hardware, the victim's CPU never runs and nothing is logged on the victim host - the sweep is silent and can run at line rate. Successful reads return the contents of whatever the victim registered; successful writes corrupt it. Memory windows (type 1/2) bind more narrowly but are seldom used, and applications that reuse a single protection domain across tenant contexts collapse the boundary entirely.

What to do

No vendor patch. Application and config work: register the smallest possible regions, never grant REMOTE_WRITE where REMOTE_READ suffices, use a separate protection domain per tenant/connection, and prefer type-2 memory windows with short lifetimes so a guessed key expires. Where the RNIC firmware supports stronger key generation, apply the vendor firmware update (firmware flash, rolling per-node, ~5-15 min per host plus a reboot on most ConnectX generations). Fabric-level containment - one P_Key or VLAN per tenant - limits who can reach the RNIC to try keys at all and is a switch config change. Budget this as an application-rearchitecture item, not a maintenance window.

References