Lightning-AI / Lightning-AI/torchmetrics

Supporting advanced metrics for image generation

Open
#1,483 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement topic: Image
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🚀 Feature

Proposal to include advanced metrics for image generation that helps to evaluate related models more completely. By completeness, I mean the following factors (as laid out in [here](https://arxiv.org/pdf/2103.01209.pdf)):

* Transparency and compositionality
* Disentanglement
* Image diversity

### Motivation

Image generation models often only use FID, IS, and KID for evaluating quality. While this has been helpful but it has shortcomings:

* FID, KID, IS tend to work better for class-conditional generation (especially where there's an alignment between classes during training the generative model and the classes on which the underlying Inception model was pre-trained)
* Therefore, these metrics might not present a complete picture of the evaluation, especially when the generation is unconditional or use free-from natural language queries (Stable Diffusion, for example).

### Pitch

Therefore, it might be helpful to approach the evaluation from many different aspects as laid out above. Here are some snapshots for each of those aspects:

#### Image diversity

image

#### Disentanglement

image

I am currently unsure how to best support these as far as API design is concerned.

### Alternatives

### Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start with the proposal and linked paper; before implementation, clarify the API and concrete metric definitions for transparency and compositionality, disentanglement, and image diversity, then establish tests for the agreed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.