Project-MONAI / Project-MONAI/MONAI
ShuffleBuffer not returning all patches
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.7k
- Forks
- 1.6k
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 20
Description
Describe the bug
While following the tutorial https://github.com/Project-MONAI/tutorials/blob/main/modules/2d_slices_from_3d_training.ipynb and implementing parts of it to my project especially when it comes to transforming Dataset containing 3D to 2D patches I encountered an issue with ShuffleBuffer.
`def create_dataset_2D_ds(ds, keys: list, trans2d: list) -> monai.data.ShuffleBuffer:
# ds = CacheDataset(data=data_dicts, transform=transforms)
patch_func = monai.data.PatchIterd(
keys=keys, patch_size=(None, None, 1), start_pos=(0, 0, 0)
)
patch_transform = Compose([
SqueezeDimd(keys=keys, dim=-1)
] + trans2d)
patch_ds = monai.data.GridPatchDataset( # -> GridPatchDataset contains 531 entries
data=ds, patch_iter=patch_func, transform=patch_transform, with_coordinates=False
)
sb = monai.data.ShuffleBuffer(patch_ds, buffer_size=200, seed=42, epochs=1) # -> ShuffleBuffer returning only 133 of them
return patch_ds`
The problem is described above in the comments, I know that the GridPatchDataset contains 531 entries, but using ShuffleBuffer in DataLoader will result in 133.
Expected behavior
Being able to Iterate with ShuffleBuffer through entire GridPatchDataset
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the tutorial notebook and the create_dataset_2D_ds function, then reproduce the 531-entry GridPatchDataset and 133-item ShuffleBuffer result through DataLoader. Inspect how GridPatchDataset, ShuffleBuffer, and the buffer_size and epochs settings interact; done means iterating through ShuffleBuffer yields all 531 patches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100