CachingBehavior.IGNORE stores metadata rows on every run, contradicting its own docstring
- Dominant language
- Jupyter Notebook
- Stars
- 2.6k
- Forks
- 213
- PR merge metrics
- No merged PRs in 30d
Description
CachingBehavior.IGNORE is documented as "Results are never stored nor versioned" (adapter.py:80), but the storage branches in `do_node_execute` (adapter.py:1281-1283) and `post_node_execute` (adapter.py:1430-1433) both lump IGNORE in with RECOMPUTE, so `_set_memory_metadata`/`_set_stored_metadata` still run for it. DISABLE is excluded from both branches; IGNORE isn't.
`pre_node_execute` returns early for IGNORE without ever calling `_set_cache_key`, so the row that gets written has `cache_key = NULL`. SQL never treats NULL as equal to NULL, so the `INSERT OR IGNORE` dedup that keeps every other node's row count at one never triggers for it. An IGNORE node gets a fresh orphaned row on every single run, forever.
Repro: a two-node DAG A -> B, `cache._ignore = ["A"]`, run it three times. `cache_metadata` ends up with 4 rows: one for B (correctly deduped) and three for A, all `cache_key = NULL`. The existing test at tests/caching/test_integration.py:496-502 (`test_ignore_behavior`) already shows this: the comment right above the assertion says "A doesn't produce any metadata or result", and the assertion two lines down is `check_metadata_store_size(cache, size=4)`, up from 2.
Looks like the fix is just dropping IGNORE from those two storage-branch tuples. Happy to send a PR if useful.
Contributor guide
Research direction
Read adapter.py:80, 1281-1283, and 1430-1433 to compare the IGNORE documentation with both metadata-storage branches. Then run tests/caching/test_integration.py::test_ignore_behavior and verify that repeated runs of the two-node DAG do not add metadata rows for node A, while the existing deduplication behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100