Project-MONAI / Project-MONAI/MONAI

ShuffleBuffer not returning all patches

Open
#7,986 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

Describe the bug
While following the tutorial https://github.com/Project-MONAI/tutorials/blob/main/modules/2d_slices_from_3d_training.ipynb and implementing parts of it to my project especially when it comes to transforming Dataset containing 3D to 2D patches I encountered an issue with ShuffleBuffer.

`def create_dataset_2D_ds(ds, keys: list, trans2d: list) -> monai.data.ShuffleBuffer:
# ds = CacheDataset(data=data_dicts, transform=transforms)
patch_func = monai.data.PatchIterd(
keys=keys, patch_size=(None, None, 1), start_pos=(0, 0, 0)
)

patch_transform = Compose([
        SqueezeDimd(keys=keys, dim=-1)
        ] + trans2d)

patch_ds = monai.data.GridPatchDataset( # -> GridPatchDataset contains 531 entries
    data=ds, patch_iter=patch_func, transform=patch_transform, with_coordinates=False
)
sb = monai.data.ShuffleBuffer(patch_ds, buffer_size=200, seed=42, epochs=1) # -> ShuffleBuffer returning only 133 of them
return patch_ds`

The problem is described above in the comments, I know that the GridPatchDataset contains 531 entries, but using ShuffleBuffer in DataLoader will result in 133.

Expected behavior
Being able to Iterate with ShuffleBuffer through entire GridPatchDataset

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the tutorial notebook and the create_dataset_2D_ds function, then reproduce the 531-entry GridPatchDataset and 133-item ShuffleBuffer result through DataLoader. Inspect how GridPatchDataset, ShuffleBuffer, and the buffer_size and epochs settings interact; done means iterating through ShuffleBuffer yields all 531 patches.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.