allenai / allenai/Lila

How to verify whether the model's output is equal to the ground truth?

未关闭
#2 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
TeX
星标
90
派生
11
PR 合并指标
30 天内没有已合并 PR

描述

Thanks for releasing the dataset of Lila benchmark.

Now, I want to evalutate the math reasoing ability by Lila benchmark.

I have a question that has been bothering me: how can we compare whether the ground truth and the model output are strictly equal?

It is difficult to compare the execution results with [output_answer]. And, It is not accurate to compare the actual execution results of the [output_program] with the execution results of our model output program.

For example:
```
Problem:
"Problem: Find the norm and argument (phase angle in radians) of $(1+i) \sqrt{10}$."
```
The gt problem is:
```
"import math from sympy import * i = 1j x = (1+i)*math.sqrt(10) norm = math.sqrt(pow(x.real, 2) + pow(x.imag, 2)) arg = math.atan(x.imag/x.real) print('Norm:', norm) print('Arg:', arg) "
```

Our model doesn't know to output the string "Norm", "Arg".

May I ask if you have any plans to release the relevant evaluation code or provide some ideas?

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。