Lightning-AI / Lightning-AI/torchmetrics
Investigate use of `join` context for distributed sync
@justusschock is already working on this.
Since Nov 15, 2022.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
## 🚀 Feature
### Motivation
Based the problems from this issue: https://github.com/Lightning-AI/metrics/issues/1297
By implementing `join` context (https://pytorch.org/tutorials/advanced/generic_join.html) for our distributed syncronization we would remove the limitation that to correctly calculate a metric the number of samples needs to be divisible by `num_gpus * batch_size` (because pytorch by default is adding additional samples to load balance).
### Pitch
Base class should derive from `Joinable` class and implement appropriate methods. It should hopefully not be too much trouble as all the sync logic is already encapsulated in a function.
### Alternatives
### Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.