Understanding Model Threats
This resource is designed to provide detailed information on various threat categories, helping you understand and mitigate potential risks in AI and machine learning systems.
Deserialization Threats
Deserialization threats occur when untrusted data or code is used to reconstruct objects, leading to potential exploitation. In AI and machine learning systems, this can result in malicious actors injecting harmful code during the deserialization process, exploiting vulnerabilities to gain unauthorized access or manipulate your systems behavior. Understanding deserialization threats is crucial for securing data integrity and preventing unauthorized code execution in your AI models.
Overview
GGUF is a binary format developed by the llama.cpp team for storing large language models so they can be loaded quickly for inference. Models are usually trained in PyTorch or another framework and then converted to GGUF. Along with the model's tensors, a GGUF file stores a standardized set of metadata.
One of these metadata fields, tokenizer.chat_template, holds a Jinja2 template that formats conversations into the prompt the model expects. Inference engines render this template every time the model is used for chat. Some, such as llama-cpp-python and Hugging Face Transformers, render it with Python's Jinja2 library. Jinja is a full templating language: if a template is rendered outside a sandboxed environment, a malicious template can break out of the template engine and run arbitrary Python code on the host (see PAIT-GGUF-101).
The scan checks the chat template for patterns that are used to escape the Jinja sandbox, such as access to __class__, __subclasses__, __globals__, __builtins__ or __import__. When a template is present but none of these patterns are found, the scan reports this issue so that you know the model ships code that will be rendered by your inference engine.
If a model reportedly has this issue, it means:
- The model is serialized using the GGUF format.
- The model contains a Jinja chat template in its
tokenizer.chat_templatemetadata field. - No known sandbox-escape or code-execution patterns were found in the template.
This is a low-severity, informational finding. Most GGUF chat models legitimately include a chat template.
Key Points
- GGUF models can store a chat template as Jinja2 code in their metadata.
- Inference engines render the chat template every time the model is used for chat.
- A malicious Jinja template rendered outside a sandbox can execute arbitrary code.
- This model's template was not found to contain malicious patterns, but it is still code supplied by the model author.
- Render chat templates in a sandboxed environment, and only load models from trusted sources.
Background Information
Further reading:
Impact
This model was not found to contain any malicious code in its chat template at this time. However, a chat template is code that your inference engine will render, so it remains an attack surface.
If a chat template were malicious and rendered without a sandbox, an attacker could:
- Access sensitive information (e.g., SSH keys, cloud credentials)
- Execute malicious code on your system
- Use the compromised system as a vector for broader attacks
How The Attack Works
Best Practices
You should:
- Only load and execute models from trusted sources
- Implement a vetting process for third-party models before use
- Use inference engines that render chat templates in a sandboxed Jinja environment, and keep them up to date
- Review the chat template, or compare it with the template published for the base model, before use
Remediation
No action is required if you trust the model's source and your inference engine renders chat templates in a sandboxed environment.
If you are not sure, compare the model's chat template with the template published by the original model creator, or replace it with a known-good template for the same model family before use.