Crawl4AI: robots.txt fetch bypasses egress controls, giving blind SSRF to metadata endpoints

Impact

The robots.txt fetch in RobotsParser.can_fetch does not apply the egress restrictions the rest of the Crawl4AI Docker server enforces. An unauthenticated caller who can set check_robots_txt can use redirects or DNS rebinding to make the crawler issue requests to loopback, internal RFC1918 and cloud metadata addresses. Crawl4AI is commonly run as a shared scraping or ingestion service next to model-serving and data pipelines, so the request originates from inside the cluster network with whatever node identity that pod carries - the classic path to an instance metadata credential or to internal admin APIs. The advisory scores it as blind SSRF for service discovery and low confidentiality impact; it does not establish credential retrieval.

Who can reach it

Anyone who can submit a crawl request to the Crawl4AI Docker server, unauthenticated. Exposure depends on whether that server is reachable from tenant networks or only from trusted pipeline code.

What to do

Upgrade Crawl4AI to 0.9.4 or later and restart the container - a service restart, no node maintenance. Additionally enforce egress at the network layer rather than in the application: block the metadata IP (169.254.169.254), require IMDSv2-style hop limits, and deny the crawler pod access to internal management subnets.

References

Related entries

• Hugging Face Accelerate: unsanitized shard paths in a checkpoint index give arbitrary file read and hangsCVE-2026-69112 · Hugging Face Accelerate (load_checkpoint_in_model / load_checkpoint_and_dispatch weight_map handling)Medium
• llama.cpp ggml RPC server: unvalidated tensor op and op_params in deserialize_tensorCVE-2026-78147 · llama.cpp ggml RPC server (deserialize_tensor op / op_params validation)Medium
• llama.cpp ggml RPC server: null pointer dereference in graph_compute kills the GPU workerCVE-2026-78148 · llama.cpp ggml RPC server (rpc_server::graph_compute)Medium
• BentoML: SSRF filter misses 100.64.0.0/10, so serving pods fetch from internal CGNAT hostsCVE-2026-78205 · BentoML make_safe_connect (SSRF address filter, RFC 6598 range)Medium
• vLLM: DeepStream backend misclassification skips pixel limits and lets unauthenticated video exhaust GPU decodeCVE-2026-78684 · vLLM (DeepStream GPU decode path, pixel-limit enforcement)Medium
• llama.cpp RPC server: crafted tensor dimensions hit a reachable assertion and abort the processCVE-2026-86317 · llama.cpp RPC server (rpc_server::deserialize_tensor in ggml/src/ggml-rpc/ggml-rpc.cpp)Medium