Lightning-AI / Lightning-AI/torchmetrics

Investigate use of `join` context for distributed sync

Open
#1,338 1 comment 0 reactions 1 assignee View on GitHub

@justusschock is already working on this.

Since Nov 15, 2022.

distributed enhancement
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🚀 Feature

### Motivation

Based the problems from this issue: https://github.com/Lightning-AI/metrics/issues/1297
By implementing `join` context (https://pytorch.org/tutorials/advanced/generic_join.html) for our distributed syncronization we would remove the limitation that to correctly calculate a metric the number of samples needs to be divisible by `num_gpus * batch_size` (because pytorch by default is adding additional samples to load balance).

### Pitch

Base class should derive from `Joinable` class and implement appropriate methods. It should hopefully not be too much trouble as all the sync logic is already encapsulated in a function.

### Alternatives

### Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.