Project-MONAI / Project-MONAI/MONAI
multiprocessing metatensor efficiency?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.7k
- Forks
- 1.6k
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 20
Description
Describe the bug
follow up of #6468, may require benchmarking https://github.com/Project-MONAI/MONAI/blob/c2a9a31beb22e1d3321016be7e68e0875cf2a8ad/monai/data/__init__.py#L121-L124
map_classes_to_indices utility (and ClassesToIndices transforms)
currently returns a list of coordinates for each class, but the type is
List[ of MetaTensors], where each class coordinate sub-list is its own
MetaTensor. This PR changes it to return a list of torch.Tensors (or
ndarray), since we don't need a MetaTensor here.I ran into an issue with current MetaTensor list, where it would just
freeze without any errors, when trying to save the cached indices
(returned from ClassesToIndices to cache, which is ListProxy shared
mem). It randomly happens, but much more frequently when the number of
classes is large (e.g. 105 output classes, so ClassesToIndices returns a
list of 105 MetaTensors). I'm not sure what the cause of the freeze is,
but my guess is that ListProxy tries to pickle each element of this list
(and struggles with MetaTensors). Disabling MetaTensor return type here,
solves the issue.
cc @myron
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin with monai/data/init.py lines 121-124 and trace map_classes_to_indices and ClassesToIndices, focusing on how their returned values are cached through multiprocessing. Benchmark the current and proposed return types, especially with many classes, and verify that saving cached indices no longer freezes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100