Hugging Face tokenizers: crafted tokenizer.json aborts the process while loading a BPE model

Impact

The builder sizes a scratch buffer to the longest vocabulary key and then writes each concatenated merge rule into it, so a merge whose concatenated token is longer than that key overruns the buffer. Rust turns the overrun into a panic that aborts the process, and because serving stacks reach this through the Python and Node FFI bindings, the abort takes the whole worker down rather than failing one request. It happens at Tokenizer::from_file/from_str time with no encoding required, so any pipeline that pulls tokenizers from a model hub, accepts tenant-supplied model directories, or shares a model cache can be crashed by whoever supplied the file. A second defect at the same location causes a usize underflow when continuing_subword_prefix is set and a merge token is shorter than the prefix: a panic in debug builds, and potential memory corruption in release builds. Observed in 0.23.1.

Who can reach it

Anyone who can place a tokenizer.json in front of the loader: a tenant uploading a model, a job pulling an unvetted repository from a model hub, or a poisoned entry in a shared model cache. No authentication to the serving process is needed, since the file itself is the input.

What to do

Upgrade the tokenizers crate or wheel past 0.23.1. The record points at the v0.23.2 tag as the branch carrying this code and does not state a fixed release, so confirm against upstream issue 2094 and the VulnCheck advisory before pinning a version as clean. Meanwhile, load tokenizer files only from sources you control and validate them out of band. Rolling the fix out means rebuilding the serving image and restarting the inference workers; no node reboot is involved.

References

Related entries

• Jupyter Server: Referer header is logged unscrubbed, leaking auth tokens into server logsCVE-2026-86049 · Jupyter Server (5xx request logging in jupyter_server/log.py)High
• n8n OpenAI Chat Model node: model-search path ignores credential domain limits and leaks the API keyCVE-2026-86082 · n8n OpenAI Chat Model node (LMChatOpenAi loadModels.ts model-search path)High
• vLLM: forged FLAC sample rate bypasses duration limit and crashes the API serverCVE-2026-90555 · vLLM (OpenAI-compatible transcription endpoint, audio header validation)High
• Hugging Face Transformers: path traversal in save_pretrained() writes files outside the save directoryCVE-2026-9856 · Hugging Face Transformers - save_pretrained() in PreTrainedTokenizerBase and ProcessorMixinHigh
• MLflow (`spark_udf` dir perms): Excessive directory permissionsCVE-2024-27134 · MLflow (`spark_udf` dir perms)High
• NVIDIA NemoClaw: code injection in the migration command gives a local user execution as the tool's privilegesCVE-2026-65082 · NVIDIA NemoClaw for Linux (migration command)High