Lightning-AI / Lightning-AI/torchmetrics
Add METEOR metric
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
## 🚀 Feature
Add another NLP metric - **METEOR** ([Lavie and Agarval, 2007](https://www.cs.cmu.edu/~alavie/METEOR/pdf/Lavie-Agarwal-2007-METEOR.pdf)).
### Motivation
METEOR is another metric used for the machine translation evaluation similarly to BLEU, however, it demonstrates a higher correlation with human judgements of translation quality.
### Pitch
To support the METEOR metric will be likely to require `nltk` dependency. The `nltk` package is also, however, needed for the ROUGE metric. METEOR thus should not bring any new dependency.
### Additional context
I will be happy to start working on this feature next week if desired to be added :)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing BLEU and ROUGE metric entry points and tests, then compare how they use the existing nltk dependency. Implement METEOR consistently with those metrics and confirm that it is available without adding a new dependency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100