Lightning-AI / Lightning-AI/torchmetrics
A way to run *any* metric async on cpu
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
🚀 Feature
Some metrics take up a lot of GPU memory in certain situations (deterministic mode, lots of classes, caching), even though they don't spit out a list (the requirement for the current "compute_on_cpu" option).
Enabling the "compute_on_cpu" option for all metrics would make this library more usable even when you have a ton of metrics that you would like to compute.
Alternatives
The issue here is that there isn't any real alternative other than not using torchmetrics or not computing some metric that you want (or spending more money on compute).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing "compute_on_cpu" option and how metric computation handles GPU state. Define the supported behavior for metrics that do not return lists, then add coverage showing that the option works across metric types without the reported GPU-memory growth.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100