intuit / intuit/fasteval

No test coverage for image generation metrics and `image-gen` extra

Open
#16 0 comments 0 reactions 1 assignee Claimed by @chotemothen View on GitHub
Dominant language
Python
Stars
12
Forks
4
PR merge metrics
No merged PRs in 30d

Description

## Description

`pyproject.toml` defines an `image-gen` optional dependency group with heavy dependencies:

```toml
image-gen = [
"Pillow>=10.0",
"transformers>=4.30",
"torch>=2.0",
]
```

However:
- There are no tests specifically covering image generation metrics (`clip_score`, `aesthetic_score`, `image_quality`, `image_faithfulness`, etc.)
- CI runs `--all-extras` which would pull in torch/transformers but doesn't validate them
- The multimodal metrics in `fasteval/metrics/multimodal.py` and `fasteval/metrics/vision.py` have no dedicated test files

## Suggested Fix

- Add test files for vision and image generation metrics (can use mocks to avoid requiring GPU/large model downloads in CI)
- Consider whether `--all-extras` in CI should exclude `image-gen` to keep CI fast, with a separate job for multimodal tests

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.