google-research / google-research/pix2struct

Bug in the relaxed_correctness eval metric

Open
#35 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
690
Forks
61
PR merge metrics
No merged PRs in 30d

Description

The relaxed_correctness metric is to compare whether the predicted answer matches with the groundtruth answer. According to its definition:

> “Following Methani et al. (2020), we use a relaxed accuracy measure for the
numeric answers to allow a minor inaccuracy that may result from the automatic
data extraction process. We consider an answer to be correct if it is within
5% of the gold answer. For non-numeric answers, we still need an exact match
to consider an answer to be correct.”

The implementation [here](https://github.com/google-research/pix2struct/blob/main/pix2struct/metrics.py#L81) seems treat the year names numeric answers, which leads to most of the year prediction questions to be correct and thus results in an incorrect high performance. For example, {pred: 2008, GT: 2010} is counted as correct because it is within 5% error, which shouldn't be the case.

Could anyone confirm this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.