Understanding Model Threats
This resource is designed to provide detailed information on various threat categories, helping you understand and mitigate potential risks in AI and machine learning systems.
Deserialization Threats
Deserialization threats occur when untrusted data or code is used to reconstruct objects, leading to potential exploitation. In AI and machine learning systems, this can result in malicious actors injecting harmful code during the deserialization process, exploiting vulnerabilities to gain unauthorized access or manipulate your systems behavior. Understanding deserialization threats is crucial for securing data integrity and preventing unauthorized code execution in your AI models.
Overview
PyTorch models saved with torch.save() (PyTorch 1.6 and later) are ZIP archives. Inside the archive, a pickle file (data.pkl) describes the model's structure, and the raw tensor data is stored in separate files under a data/ directory. For each tensor storage, the pickle records which data file holds its bytes and how many elements the storage contains.
When the model is loaded, PyTorch allocates each storage based on the size declared in the pickle and reads the matching data file into it. If the declared size does not match the data file, a crafted model can make PyTorch read or write outside the memory it allocated. This kind of memory corruption can be used to achieve arbitrary code execution. It does not depend on the pickle calling dangerous functions, so loading with torch.load(..., weights_only=True), which blocks dangerous pickle functions and is commonly assumed to make loading untrusted PyTorch files safe, does not prevent it.
The scan compares the storage size declared for each tensor in the model's pickle with the actual size of its data file in the archive.
If a model reportedly has this issue, it means:
- The model is serialized using the PyTorch ZIP format (PyTorch 1.6 and later).
- A tensor storage in the model declares more elements than its data file contains, or declares a negative size.
A legitimately saved PyTorch model never declares a storage larger than its data, so this mismatch indicates a deliberately crafted file.
Key Points
- PyTorch model files store each tensor's declared size separately from its raw data.
- A declared size that is larger than the actual data can corrupt memory when the model is loaded.
- Memory corruption during loading can lead to arbitrary code execution.
weights_only=Trueonly restricts which functions the pickle can call, so it does not protect against this attack.- Only load models from trusted sources.
Background Information
Further reading:
Impact
An attacker could exploit a compromised model to:
- Execute malicious code on your system, even when the model is loaded with
weights_only=True - Access sensitive information in process memory or on the system (e.g., SSH keys, cloud credentials)
- Crash the loading process, causing a denial of service
- Use the compromised system as a vector for broader attacks
Note: Because this attack does not rely on dangerous functions in the pickle, scanners and loaders that only check for unsafe pickle operations will not detect it.
How The Attack Works
Best Practices
You should:
- Only load and execute models from trusted sources
- Implement a vetting process for third-party models before use
- Use sandboxing techniques when loading untrusted models
- Do not rely on
weights_only=Truealone to make untrusted PyTorch files safe - Use model formats that don't use pickle, such as SafeTensors, which provides a safe alternative to traditional serialization formats
Remediation
Do not load this model. A storage size mismatch does not occur in models saved normally by PyTorch, and the file cannot be repaired reliably without knowing what the original tensors were.
If possible, obtain the model in a different format like SafeTensors, or from a trusted source.
If not possible, reach out to the model creator and alert them that the model has failed our scan. You can even link to the specific page on our Insights Database to provide our most up to date findings.
The model provider should also report what they did to correct this issue as part of their release notes.