Envoy: large requests crash the TCP connection pool via a null buffer watermark callback

Impact

A crash in the Envoy proxy process takes down whatever traffic it fronts. On GPU fleets Envoy is commonly the ingress and the service-mesh sidecar in front of inference endpoints, so a dereference of a null buffer watermark callback during connection teardown drops in-flight inference requests and, for sidecar deployments, can be triggered per pod by whoever can send a large request or elicit a large response. The flaw needs no authentication and reaches both TCP proxy and mixed HTTP/1-HTTP/2 ALPN configurations. Availability only - there is no indication of memory disclosure or code execution.

Who can reach it

Anyone who can reach an Envoy listener, unauthenticated, and send a large request or cause a large upstream response while the connection is closing.

What to do

Upgrade to Envoy 1.36.1, 1.35.5, 1.34.9 or 1.33.10. Rolling restart of the proxy fleet; for sidecar mesh deployments this means a rollout that restarts pods, which on GPU workloads should be sequenced with job boundaries rather than forced mid-training.

References

Related entries

• Argo CD (Helm OCI repository credential logging): CREDENTIAL DISCLOSURE THROUGH THE LOG PIPELINE: Argo CD wrote theNCVD-2021-016-argo-cd-helm-oci-repository-cred · Argo CD (Helm OCI repository credential logging)Medium
• Helm: Path traversal in `helm fetch --untar` writes outside the target directoryCVE-2019-1000008 · HelmMedium
• Kubernetes (kubectl): `kubectl cp` path traversal from a malicious container tar overwrites files on the operator'sCVE-2019-11246 · Kubernetes (kubectl)Medium
• Kubernetes (kubectl): Follow-up incomplete fix for the kubectl cp traversalCVE-2019-11249 · Kubernetes (kubectl)Medium
• Kubernetes (client-go): Bearer tokens logged at verbosity 7+CVE-2019-11250 · Kubernetes (client-go)Medium
• Kubernetes (kube-apiserver): YAML parsing CPU exhaustion in the apiserverCVE-2019-11254 · Kubernetes (kube-apiserver)Medium