Project-MONAI / Project-MONAI/MONAI
Load data repeatedly in class CSVIterableDataset from monai/data/iterable_dataset.py
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.7k
- Forks
- 1.6k
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 20
Description
Describe the bug
In iterable_dataset.py
def __iter__(self):
if self.shuffle:
self.seed += 1
buffer = ShuffleBuffer(
data=self._flattened(), transform=self.transform, buffer_size=self.buffer_size, seed=self.seed
)
yield from buffer
yield from IterableDataset(data=self._flattened(), transform=self.transform)
This may cause loading data repeatedly, and maybe the code should be
def __iter__(self):
if self.shuffle:
self.seed += 1
buffer = ShuffleBuffer(
data=self._flattened(), transform=self.transform, buffer_size=self.buffer_size, seed=self.seed
)
yield from buffer
else:
yield from IterableDataset(data=self._flattened(), transform=self.transform)
If I understand the logic in the code incorrectly, please point it out directly, thanks a lot.
To Reproduce
Steps to reproduce the behavior:
- Go to '...'
- Install '....'
- Run commands '....'
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment
Ensuring you use the relevant python executable, please paste the output of:
python -c "import monai; monai.config.print_debug_info()"
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in monai/data/iterable_dataset.py at CSVIterableDataset.iter and compare the shuffle and non-shuffle paths shown in the report. Reproduce the iteration behavior and verify that data is not loaded or yielded repeatedly when shuffling is enabled; the issue is done when the intended single path is confirmed and covered by an appropriate check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100