How to verify whether the model's output is equal to the ground truth?
- 主要语言
- TeX
- 星标
- 90
- 派生
- 11
- PR 合并指标
- 30 天内没有已合并 PR
描述
Thanks for releasing the dataset of Lila benchmark.
Now, I want to evalutate the math reasoing ability by Lila benchmark.
I have a question that has been bothering me: how can we compare whether the ground truth and the model output are strictly equal?
It is difficult to compare the execution results with [output_answer]. And, It is not accurate to compare the actual execution results of the [output_program] with the execution results of our model output program.
For example:
```
Problem:
"Problem: Find the norm and argument (phase angle in radians) of $(1+i) \sqrt{10}$."
```
The gt problem is:
```
"import math from sympy import * i = 1j x = (1+i)*math.sqrt(10) norm = math.sqrt(pow(x.real, 2) + pow(x.imag, 2)) arg = math.atan(x.imag/x.real) print('Norm:', norm) print('Arg:', arg) "
```
Our model doesn't know to output the string "Norm", "Arg".
May I ask if you have any plans to release the relevant evaluation code or provide some ideas?
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。