Project-MONAI / Project-MONAI/MONAI
GridPatchDataset caching can repeat, drop, or reject samples
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.7k
- Forks
- 1.6k
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 20
Description
Describe the bug
GridPatchDataset(cache=True) does not preserve dataset contents for several documented cache configurations:
- With a partial cache (
0 < cache_rate < 1), an uncached item following a cached item reuses the previous item's cache index and emits the cached patches again. - With
with_coordinates=False, cached patches are zipped with an empty coordinate cache, so no cached patches are emitted. - With
cache_rate=0(orcache_num=0), initialization tries to unpack an empty cache and raisesValueError. - With a transform pipeline containing no random transform, cached iteration passes
start=NonetoComposeand raisesValueError.
These are data-correctness issues: caching can silently replace or drop training samples, or reject valid cache configurations.
To Reproduce
from monai.data import GridPatchDataset
from monai.transforms import Lambda
def patches(image):
for item in image:
yield item, item * 10
partial = GridPatchDataset(
[[1], [2]], patches, cache=True, cache_rate=0.5, progress=False
)
print(list(partial))
# Actual: [(1, 10), (1, 10)]
# Expected: [(1, 10), (2, 20)]
no_coordinates = GridPatchDataset(
[[1, 2]], patches, with_coordinates=False, cache=True, progress=False
)
print(list(no_coordinates))
# Actual: []
# Expected: [1, 2]
zero_cache = GridPatchDataset(
[[1], [2]], patches, cache=True, cache_rate=0, progress=False
)
# Actual: ValueError: not enough values to unpack (expected 2, got 0)
# Expected: construct successfully and iterate without using a cache
deterministic = GridPatchDataset(
[[1]], patches, transform=Lambda(lambda x: x + 100), cache=True, progress=False
)
print(list(deterministic))
# Actual: ValueError: 'start' (None) cannot be None
# Expected: [(101, 10)]
Expected behavior
Enabling caching must not change which patches are yielded. cache_rate should only select how many source items are cached, and with_coordinates should only control whether coordinates are included in each yielded item. Deterministic transforms should be computed while populating the cache, and a cache hit should resume at the end of that transform pipeline.
Environment
MONAI version: 0+untagged.3487.gd1306f6
MONAI rev id: d1306f6d1996cffeb9d10984dd1c056b7fe2ed1d
Python version: 3.12.0
NumPy version: 2.5.2
PyTorch version: 2.14.0+cpu
OS: Windows
Additional context
The partial-cache issue comes from cache_index being initialized before the image loop and not reset for cache misses. The coordinate-free path always calls zip(data, other) even though _cache_other is intentionally empty when with_coordinates=False. set_data() unconditionally unpacks zip(*self._fill_cache(...)), including when the configured cache size is zero. Finally, Compose.get_index_of_first(...) returns None when every transform is deterministic, but the cache-read path uses that value as the start index.
The cache implementation was introduced in #7180. Existing coverage exercises a full cache with coordinates enabled and a pipeline containing a random transform, so these paths are currently untested.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with GridPatchDataset.set_data() and _fill_cache(), then inspect the iteration path and Compose.get_index_of_first(). Run the four reproductions in the issue, and add coverage for partial, coordinate-free, zero-size, and deterministic-transform caches; done means cached and uncached iteration yield the same patches without initialization or transform errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100