RoCEv2 congestion control - DCQCN, ECN marking and Congestion Notification Packets: DCQCN reacts to ECN marks by having

Impact

DCQCN reacts to ECN marks by having the receiver send Congestion Notification Packets that make the sender cut its rate. CNPs are unauthenticated RoCE packets like any other, so an attacker who can spoof onto the fabric can forge CNPs at a victim's sender and drive its rate to the floor while their own traffic is unaffected - a targeted bandwidth-theft and starvation primitive. Conversely, a tenant whose NIC simply ignores CNPs takes an unfair share and pushes everyone else into PFC. Published measurement shows the native PFC-based scheme already suffers unfairness and head-of-line blocking, and that congestion-control choice materially changes distributed DNN training time, so the manipulation lands directly on job completion times in a GPU cluster.

Who can reach it

Forged CNPs need only a spoofed source GID and the victim's QP number - the same predictability that makes packet injection work. Rate-ignoring is even simpler: run a NIC configuration or a custom firmware/driver that under-responds to congestion notifications, which looks like a tuning choice rather than an attack. Both are invisible to host-level monitoring; they show up only as unexplained throughput asymmetry between tenants.

What to do

Config change: enforce switch-side per-tenant rate limiting and ECN marking policy rather than trusting endpoint congestion response, apply source-address filtering so CNPs cannot be spoofed across tenants, and keep tenants in separate traffic classes so an unresponsive one cannot starve others. Standardise and lock the DCQCN parameter set through the NIC driver configuration (mlxconfig / sysfs) so tenants cannot retune their own NICs - a driver-level config change applied at provisioning, no reboot. Per-tenant switch queues are the durable fix and may require a QoS profile change plus a switch reload on constrained platforms. Monitor per-QP CNP counts as a detection signal.

References

Related entries

• RNIC microarchitectural resources (NIC cache, processing units) under multi-tenant RDMA: This is the paper thatNCVD-2023-006-rnic-microarchitectural-resource · RNIC microarchitectural resources (NIC cache, processing units) under multi-tenant RDMAMedium
• RNIC microarchitectural resources (NIC cache, processing units) under multi-tenant RDMA: This is the paper thatNCVD-2023-008-rnic-microarchitectural-resource · RNIC microarchitectural resources (NIC cache, processing units) under multi-tenant RDMAMedium
• TPM 2.0 (S3 sleep PCR reset): Platform Configuration Registers can be reset without a full platform restart by abusingCVE-2018-6622 · TPM 2.0 (S3 sleep PCR reset)Medium
• GRUB2 (grub_malloc allocator): GRUB's allocator never checks the requested size for arithmetic overflow, so a tenantCVE-2020-14308 · GRUB2 (grub_malloc allocator)Medium
• GRUB2 (squashfs symlink parser): Integer overflow in grub_squash_read_symlink lets a crafted squashfs image driveCVE-2020-14309 · GRUB2 (squashfs symlink parser)Medium
• GRUB2 (read_section_from_string): Integer overflow while reading a section string overflows the heap and gives controlCVE-2020-14310 · GRUB2 (read_section_from_string)Medium