NVIDIA-Merlin / NVIDIA-Merlin/Transformers4Rec

[SECURITY] Pickle allowlist in `utils/serialization.py` is bypassable (`builtins.getattr` allowed, duplicate keys drop `PredictionTask`)

Open
#814 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
165
Avg merge
1m
Merged PRs (30d)
2

Description

Bug description

The BASE_SERIALIZATION_CLASSES allowlist introduced by PR #802 (commit ab7207cf, "Sec pic fix") in transformers4rec/utils/serialization.py has two structural issues that defeat the intended restriction on Unpickler.find_class.

File (at ab7207cf):
https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/ab7207cf40c7960f7aa22c86ab232576aa8cf847/transformers4rec/utils/serialization.py#L11-L68

The file was subsequently removed in PR #808 (41b14d7b), but checkpoints produced during the #802 → #807 window still exist in the wild and the design questions apply to any follow-up allowlist.

(1) builtins key appears twice. Python dict literals are last-write-wins, so only the second declaration survives:

BASE_SERIALIZATION_CLASSES = {
    "builtins": [
        "Exception", "ValueError", "NotImplementedError", "AttributeError",
        "AssertionError"
    ],
    ...
    "builtins": ["getattr"],   # <-- this overrides the first "builtins" key
}

Net result at runtime: builtins maps to ["getattr"]. The Exception subclasses (intentionally allowlisted) are silently dropped, and builtins.getattr — a well-known primitive in pickle gadget chains (attribute traversal → code execution) — is approved.

(2) torch.storage._load_from_bytes is in the allowlist. _load_from_bytes wraps torch.load, which itself performs unrestricted pickle deserialization when the installed PyTorch version predates the weights_only=True default. Combined with getattr, this provides a reachable path from the restricted unpickler to arbitrary code execution.

(3) transformers4rec.torch.model.base is also duplicated. The first declaration includes PredictionTask; the second (which wins) does not:

"transformers4rec.torch.model.base": ["Model", "Head", "PredictionTask"],
...
"transformers4rec.torch.model.base": ["forward_to_prediction_fn", "Model", "Head"],   # wins; PredictionTask dropped

So any checkpoint containing PredictionTask fails to deserialize with ValueError from the restricted unpickler. This is an availability regression, not a security one, but it indicates the dict literal was not reviewed carefully before merge.

Steps/Code to reproduce bug

Structural evidence (no exploit payload):

git show ab7207cf:transformers4rec/utils/serialization.py | grep -n '"builtins"'
# 12:    "builtins": [
# 66:    "builtins": ["getattr"],

git show ab7207cf:transformers4rec/utils/serialization.py | grep -n 'torch.storage'
# 49:    "torch.storage": ["_load_from_bytes"],

Evaluating the module confirms the surviving values:

import ast
src = open(".../serialization.py").read()
ns = {}
exec(src, ns)
print(ns["BASE_SERIALIZATION_CLASSES"]["builtins"])            # ['getattr']
print(ns["BASE_SERIALIZATION_CLASSES"]["torch.storage"])       # ['_load_from_bytes']
print("PredictionTask" in ns["BASE_SERIALIZATION_CLASSES"]
      ["transformers4rec.torch.model.base"])                    # False

I am intentionally not attaching an end-to-end exploit payload to a public issue; the combination of builtins.getattr + torch.storage._load_from_bytes is sufficient for any reader familiar with pickle gadget chains to reproduce.

Expected behavior
  • The allowlist does not contain builtins.getattr or torch.storage._load_from_bytes.
  • The dict has no duplicate keys (a flake8-bugbear B033 or similar lint would have caught this).
  • The design shifts away from pickle-based serialization for model weights: either safetensors, or torch.load(..., weights_only=True) after bumping the minimum PyTorch version (see the companion torch.load issue I am filing).
Environment details
  • Transformers4Rec: commits ab7207cf (PR #802) through 0e31f575 (PR #807); file removed at 41b14d7b (PR #808) but affected checkpoints persist.
  • Python: any
  • PyTorch: any (the _load_from_bytes gadget exists across all modern versions)
Additional context

Suggested remediation order:

  1. Restore allowlist-based protection only as a temporary measure — remove builtins.getattr and torch.storage._load_from_bytes, de-duplicate all keys.
  2. Transition model-weight serialization to a format that does not execute code on load (safetensors is the common choice for HF-adjacent projects).
  3. For any remaining pickle-based artifacts, wrap torch.load with an explicit weights_only=True (see the companion issue).

I can prepare separate PRs for each step if the maintainers prefer.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the historical transformers4rec/utils/serialization.py from commit ab7207cf and compare it with the current serialization paths, since the file was removed in PR #808. Verify the duplicate-key behavior and the allowed torch.storage entry, then review the companion torch.load issue. Done means no unsafe entries or duplicate keys remain and a safe replacement path is defined for affected checkpoints.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.