Hugging Face tokenizers: crafted tokenizer.json aborts the process while loading a BPE model

Impact

The builder sizes a scratch buffer to the longest vocabulary key and then writes each concatenated merge rule into it, so a merge whose concatenated token is longer than that key overruns the buffer. Rust turns the overrun into a panic that aborts the process, and because serving stacks reach this through the Python and Node FFI bindings, the abort takes the whole worker down rather than failing one request. It happens at Tokenizer::from_file/from_str time with no encoding required, so any pipeline that pulls tokenizers from a model hub, accepts tenant-supplied model directories, or shares a model cache can be crashed by whoever supplied the file. A second defect at the same location causes a usize underflow when continuing_subword_prefix is set and a merge token is shorter than the prefix: a panic in debug builds, and potential memory corruption in release builds. Observed in 0.23.1.

Who can reach it

Anyone who can place a tokenizer.json in front of the loader: a tenant uploading a model, a job pulling an unvetted repository from a model hub, or a poisoned entry in a shared model cache. No authentication to the serving process is needed, since the file itself is the input.

What to do

Upgrade the tokenizers crate or wheel past 0.23.1. The record points at the v0.23.2 tag as the branch carrying this code and does not state a fixed release, so confirm against upstream issue 2094 and the VulnCheck advisory before pinning a version as clean. Meanwhile, load tokenizer files only from sources you control and validate them out of band. Rolling the fix out means rebuilding the serving image and restarting the inference workers; no node reboot is involved.

References

Related entries

• Hugging Face Transformers: path traversal in save_pretrained() writes files outside the save directoryCVE-2026-9856 · Hugging Face Transformers - save_pretrained() in PreTrainedTokenizerBase and ProcessorMixinHigh
• MLflow (`spark_udf` dir perms): Excessive directory permissionsCVE-2024-27134 · MLflow (`spark_udf` dir perms)High
• NVIDIA NemoClaw: code injection in the migration command gives a local user execution as the tool's privilegesCVE-2026-65082 · NVIDIA NemoClaw for Linux (migration command)High
• llama.cpp ggml RPC server: unvalidated tensor op and op_params in deserialize_tensorCVE-2026-78147 · llama.cpp ggml RPC server (deserialize_tensor op / op_params validation)Medium
• llama.cpp ggml RPC server: null pointer dereference in graph_compute kills the GPU workerCVE-2026-78148 · llama.cpp ggml RPC server (rpc_server::graph_compute)Medium
• BentoML: SSRF filter misses 100.64.0.0/10, so serving pods fetch from internal CGNAT hostsCVE-2026-78205 · BentoML make_safe_connect (SSRF address filter, RFC 6598 range)Medium