Prometheus: unvalidated snappy decoded length on /api/v1/read lets a small request exhaust server memory

Impact

The remote read endpoint allocates memory based on the decoded length declared in a snappy-compressed request body without checking it against anything, so a tiny unauthenticated request triggers a very large heap allocation. Sent concurrently, this exhausts memory and kills the Prometheus process - and on a node with a container memory limit it will be the OOM killer that does it. For a GPU fleet this takes out the metrics path that alerting, capacity planning and per-tenant GPU accounting run on, and a Prometheus that is being repeatedly crashed will not stay up long enough to catch up on scrapes. Only servers with the remote read endpoint reachable are exposed.

Who can reach it

Anyone who can send an HTTP request to /api/v1/read on the Prometheus server. No authentication is required.

What to do

Upgrade to Prometheus 3.5.3 or 3.11.3 (or the corresponding Red Hat errata) and restart the process. If a patch window is not immediate, put the remote read endpoint behind an authenticating proxy or firewall it off from anything but the intended readers; disabling remote read entirely is the strongest stopgap where nothing depends on it.

References