Lightning-AI / Lightning-AI/torchmetrics
Supporting advanced metrics for image generation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
## 🚀 Feature
Proposal to include advanced metrics for image generation that helps to evaluate related models more completely. By completeness, I mean the following factors (as laid out in [here](https://arxiv.org/pdf/2103.01209.pdf)):
* Transparency and compositionality
* Disentanglement
* Image diversity
### Motivation
Image generation models often only use FID, IS, and KID for evaluating quality. While this has been helpful but it has shortcomings:
* FID, KID, IS tend to work better for class-conditional generation (especially where there's an alignment between classes during training the generative model and the classes on which the underlying Inception model was pre-trained)
* Therefore, these metrics might not present a complete picture of the evaluation, especially when the generation is unconditional or use free-from natural language queries (Stable Diffusion, for example).
### Pitch
Therefore, it might be helpful to approach the evaluation from many different aspects as laid out above. Here are some snapshots for each of those aspects:
#### Image diversity

#### Disentanglement

I am currently unsure how to best support these as far as API design is concerned.
### Alternatives
### Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start with the proposal and linked paper; before implementation, clarify the API and concrete metric definitions for transparency and compositionality, disentanglement, and image diversity, then establish tests for the agreed behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100