apache / apache/hamilton

CachingBehavior.IGNORE stores metadata rows on every run, contradicting its own docstring

Open Beginner friendly
#1,704 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

CachingBehavior.IGNORE is documented as "Results are never stored nor versioned" (adapter.py:80), but the storage branches in `do_node_execute` (adapter.py:1281-1283) and `post_node_execute` (adapter.py:1430-1433) both lump IGNORE in with RECOMPUTE, so `_set_memory_metadata`/`_set_stored_metadata` still run for it. DISABLE is excluded from both branches; IGNORE isn't.

`pre_node_execute` returns early for IGNORE without ever calling `_set_cache_key`, so the row that gets written has `cache_key = NULL`. SQL never treats NULL as equal to NULL, so the `INSERT OR IGNORE` dedup that keeps every other node's row count at one never triggers for it. An IGNORE node gets a fresh orphaned row on every single run, forever.

Repro: a two-node DAG A -> B, `cache._ignore = ["A"]`, run it three times. `cache_metadata` ends up with 4 rows: one for B (correctly deduped) and three for A, all `cache_key = NULL`. The existing test at tests/caching/test_integration.py:496-502 (`test_ignore_behavior`) already shows this: the comment right above the assertion says "A doesn't produce any metadata or result", and the assertion two lines down is `check_metadata_store_size(cache, size=4)`, up from 2.

Looks like the fix is just dropping IGNORE from those two storage-branch tuples. Happy to send a PR if useful.

Contributor guide

Open the contributing guide

Research direction

Read adapter.py:80, 1281-1283, and 1430-1433 to compare the IGNORE documentation with both metadata-storage branches. Then run tests/caching/test_integration.py::test_ignore_behavior and verify that repeated runs of the two-node DAG do not add metadata rows for node A, while the existing deduplication behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.