Project-MONAI / Project-MONAI/MONAI

Load data repeatedly in class CSVIterableDataset from monai/data/iterable_dataset.py

Open
#7,869 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

Describe the bug
In iterable_dataset.py

def __iter__(self):
    if self.shuffle:
        self.seed += 1
        buffer = ShuffleBuffer(
            data=self._flattened(), transform=self.transform, buffer_size=self.buffer_size, seed=self.seed
        )
        yield from buffer
    yield from IterableDataset(data=self._flattened(), transform=self.transform)

This may cause loading data repeatedly, and maybe the code should be

def __iter__(self):
    if self.shuffle:
        self.seed += 1
        buffer = ShuffleBuffer(
            data=self._flattened(), transform=self.transform, buffer_size=self.buffer_size, seed=self.seed
        )
        yield from buffer
   else:
        yield from IterableDataset(data=self._flattened(), transform=self.transform)

If I understand the logic in the code incorrectly, please point it out directly, thanks a lot.
To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Install '....'
  3. Run commands '....'

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c "import monai; monai.config.print_debug_info()"

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in monai/data/iterable_dataset.py at CSVIterableDataset.iter and compare the shuffle and non-shuffle paths shown in the report. Reproduce the iteration behavior and verify that data is not loaded or yielded repeatedly when shuffling is enabled; the issue is done when the intended single path is confirmed and covered by an appropriate check.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.