[Performance] ModelPatcher creates a new namedtuple class for every weight backup
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
## Expected Behavior
`ModelPatcher.backup` entries should reuse one tuple type. For `N` backed-up weights:
```python
len(self.backup) == N
len({type(entry) for entry in self.backup.values()}) == 1
```
Creating and clearing backup entries should not generate a new class and its associated cyclic objects for every weight.
## Actual Behavior
`comfy/model_patcher.py` calls `collections.namedtuple()` inside two per-key hot paths. Because `collections.namedtuple()` creates a new class on every call, ComfyUI creates one distinct `Dimension` type per backed-up weight:
```python
self.backup[key] = collections.namedtuple(
"Dimension", ["weight", "inplace_update"]
)(weight.to(device=self.offload_device, copy=inplace_update), inplace_update)
```
and:
```python
self.backup[key] = collections.namedtuple(
"Dimension", ["weight", "inplace_update"]
)(weight, False)
```
Therefore:
```python
len({type(entry) for entry in self.backup.values()}) == len(self.backup)
```
Each generated class brings its own methods, closure cells, descriptors, `staticmethod`, `classmethod`, and two `_tuplegetter` objects. Repeated model patch/load/unload cycles generate large amounts of class-shaped cyclic garbage.
In one 178.1-second production observation window, the GC-tracked heap grew with the exact signature of approximately 925 newly generated two-field namedtuple classes:
```text
total tracked +65,105
type +931
staticmethod +925
classmethod +925
_tuplegetter +1,850
function +5,568
cell +8,365
tuple +23,049
frozenset +17,550
Gen 2 collections: +13
Objects collected by Gen 2: +310,277
Average per Gen 2: ~23,867
```
In an isolated reproduction using the actual upstream assignment, 2,000 backup entries created 2,000 distinct classes. Clearing the backup made Gen 2 collect approximately 58,000–62,000 class-related cyclic objects, depending on the Python version. The shared-type implementation created one class and no per-entry class-related cyclic garbage.
## Steps to Reproduce
This reproduces the core allocation behavior on a clean ComfyUI checkout without loading custom nodes or requiring a workflow/model file:
1. Check out current ComfyUI `master` and activate its Python environment.
2. Save the following as `/tmp/reproduce_model_patcher_namedtuple_gc.py`:
```python
import gc
import torch
from comfy.model_patcher import ModelPatcher
model = torch.nn.Module()
keys = []
for index in range(1000):
key = f"weight_{index}"
keys.append(key)
model.register_parameter(
key,
torch.nn.Parameter(torch.ones(1)),
)
patcher = ModelPatcher(
model,
load_device=torch.device("cpu"),
offload_device=torch.device("cpu"),
)
# Exclude the interpreter/import baseline from the cleanup measurement.
gc.collect()
gc.freeze()
gc.disable()
for key in keys:
patcher.patch_weight_to_device(key, force_cast=True)
print("backup entries:", len(patcher.backup))
print(
"unique backup types:",
len({type(entry) for entry in patcher.backup.values()}),
)
patcher.backup.clear()
print("gen2 collected after clear:", gc.collect(2))
```
3. Run it from the ComfyUI repository root:
```bash
python /tmp/reproduce_model_patcher_namedtuple_gc.py
```
4. Current upstream behavior is approximately:
```text
backup entries: 1000
unique backup types: 1000
gen2 collected after clear: ~29,000-31,000
```
5. With the proposed shared module-level type:
```text
backup entries: 1000
unique backup types: 1
gen2 collected after clear: 0 class-related objects
```
The issue is in ComfyUI core and the clean reproduction does not import custom nodes. The production workload did contain custom nodes, but they are not needed to reproduce the class-per-backup behavior.
## Debug Logs
These production logs motivated the investigation. They demonstrate the symptom but, as explained under **Other**, subsequent testing found an independent execution-cache factor; the absolute pause duration should not be attributed solely to the inline `namedtuple` calls.
Pre-fix examples:
```text
2026-08-11 09:40:58.853 | WARNING | req_id:- | hooks.gc_trace:_gc_callback:73 - slow gc_trace | gen:2 cost:15379.6ms collected:26200 uncollectable:0 counts:(1, 0, 0)
2026-08-11 15:03:54.158 | WARNING | req_id:- | hooks.gc_trace:_gc_callback:73 - slow gc_trace | gen:2 cost:17517.8ms collected:31913 uncollectable:0 counts:(2, 0, 0)
```
Summary of 3,495 consecutive pre-fix Gen 2 collections over 24 hours:
```text
time range: 2026-08-10 11:03:08.072 -> 2026-08-11 11:02:54.467
samples: 3495
min: 6498.3 ms
mean: 7351.1 ms
median: 7344.7 ms
p90: 7954.3 ms
p95: 8033.6 ms
p99: 8183.6 ms
max: 8455.0 ms
max collected: 47808
uncollectable total: 0
```
The hourly median increased from 6,636.4 ms to 8,042.6 ms during that 24-hour log.
Post-fix examples from a newly started worker:
```text
2026-08-12 10:34:42.815 | WARNING | req_id:- | hooks.gc_trace:_gc_callback:73 - slow gc_trace | gen:2 cost:485.3ms collected:948 uncollectable:0 counts:(1, 0, 0)
2026-08-12 10:40:14.053 | WARNING | req_id:- | hooks.gc_trace:_gc_callback:73 - slow gc_trace | gen:2 cost:493.5ms collected:996 uncollectable:0 counts:(1, 0, 0)
```
Summary of 100 consecutive post-fix collections:
```text
samples: 100
min: 437.3 ms
median: 480.1 ms
p95: 513.0 ms
max: 520.1 ms
last: 517.9 ms
max collected: 1313
```
This production before/after comparison is observational rather than a controlled benchmark because deploying the fix also restarted the worker and reset its execution cache. The controlled evidence for this issue is the class count and isolated GC-object count in **Steps to Reproduce** and **Other**.
The GC timer uses `gc.callbacks` and measures wall time between the `start` and `stop` phases with `time.perf_counter()`. The `collected` and `uncollectable` values come directly from the callback's `info` argument.
## Other
### Upstream revision used for reproduction
The two inline allocations were verified on upstream commit `27bca654eb9a70237d93f56a6ea336ab55f8925d` on 2026-08-12:
- https://github.com/Comfy-Org/ComfyUI/blob/27bca654eb9a70237d93f56a6ea336ab55f8925d/comfy/model_patcher.py#L907
- https://github.com/Comfy-Org/ComfyUI/blob/27bca654eb9a70237d93f56a6ea336ab55f8925d/comfy/model_patcher.py#L1976
The issue was originally validated against commit `62b3c94bd45154f6486c7abf1b9efcacee96ea69` and reproduced on Python 3.9.6, 3.11.15, and 3.12.13.
### Investigation update: independent execution-cache growth
Further investigation found a separate source of full-GC growth in the production service. The default RAM-pressure execution cache retained input-signature keys composed primarily of tuples and frozensets until memory pressure caused eviction.
On the same worker running the shared-type fix, the execution cache was allowed to grow and was then cleared through ComfyUI's `/free` endpoint:
| Metric | Before clearing execution cache | After clearing execution cache |
|---|---:|---:|
| GC-tracked objects | 1,839,066 | 152,673 |
| `tuple` | 876,414 | 35,960 |
| `frozenset` | 873,963 | 28,355 |
| Gen 2 GC duration | approximately 624 ms | 44.9 ms |
The removed tuples and frozensets accounted for 1,686,062 of the 1,686,393-object decrease. Meanwhile, the tracked `type` count remained stable at 887 across repeated post-fix workflows, and `_tuplegetter` showed no growth. This confirms that the per-backup class creation stopped, while also showing that execution-cache state independently affects absolute full-GC duration.
Consequently, the earlier production before/after timing cannot be used to claim that the inline `namedtuple` construction alone caused the 15–17 second pauses. This report is intentionally limited to the independently reproducible class-per-backup allocation and its associated cyclic-GC churn.
### Proposed fix
Create the type once at module scope while preserving the existing tuple behavior, typename, and `.weight` / `.inplace_update` attributes:
```python
ModelPatcherBackup = collections.namedtuple(
"Dimension",
["weight", "inplace_update"],
)
```
Then use it at both call sites:
```python
self.backup[key] = ModelPatcherBackup(
weight.to(
device=self.offload_device,
copy=inplace_update,
),
inplace_update,
)
```
and:
```python
self.backup[key] = ModelPatcherBackup(weight, False)
```
A regression test creates two backup entries through `patch_weight_to_device()` and asserts that both are instances of `ModelPatcherBackup` and share exactly one type.
### Additional diagnostics
An isolated benchmark on the affected production Python 3.11.15 build produced the following results:
| Backup entries | Implementation | Unique types | Live Gen 2 GC | Cleanup Gen 2 GC | Cleanup collected |
|---:|---|---:|---:|---:|---:|
| 925 | Current source | 925 | 3.03 ms | 5.60 ms | 26,825 |
| 925 | Shared type | 1 | 0.04 ms | 0.00 ms | 0 |
| 2,000 | Current source | 2,000 | 7.53 ms | 17.01 ms | 58,000 |
| 2,000 | Shared type | 1 | 0.08 ms | 0.00 ms | 0 |
| 10,000 | Current source | 10,000 | 54.80 ms | 103.69 ms | 290,000 |
| 10,000 | Shared type | 1 | 0.36 ms | 0.00 ms | 0 |
The production log's common collected-object counts closely match the benchmark's approximately 29 cyclic objects per generated class. For example, 925 generated types produced 26,825 collectible objects.
The production deployment was followed by a large reduction in Gen 2 duration, but that comparison also included a process restart and execution-cache reset. It therefore cannot isolate the timing improvement caused by this change. The controlled benchmark establishes the narrower claim: the current implementation creates one class and approximately 29 cyclic objects per backup entry, whereas the shared-type implementation eliminates that per-entry class allocation and associated cyclic garbage.
A `/proc` sampler and a `py-spy --gil --native` profile found the observed slow collections CPU-bound in CPython's `gc_collect_main`, without cgroup throttling, memory/IO pressure, or Python-level model finalizers.
Environment:
```text
Python: 3.11.15
PyTorch: 2.8.0+cu128
OS: Linux container
Pre-fix worker uptime at initial observation: approximately 6 days
cpu.cfs_quota_us: -1
nr_throttled: 0
throttled_time: 0
swap: 0
oom_kill: 0
```
Contributor guide
Research direction
Read comfy/model_patcher.py at the two namedtuple call sites around lines 907 and 1976, then inspect patch_weight_to_device() and the backup handling. Add the regression test described in the issue, run the clean reproduction or relevant test suite, and verify backup entries share one type without changing their attributes or behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100