How to verify whether the model's output is equal to the ground truth?
- Lingua principale
- TeX
- Stelle
- 90
- Fork
- 11
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
Thanks for releasing the dataset of Lila benchmark.
Now, I want to evalutate the math reasoing ability by Lila benchmark.
I have a question that has been bothering me: how can we compare whether the ground truth and the model output are strictly equal?
It is difficult to compare the execution results with [output_answer]. And, It is not accurate to compare the actual execution results of the [output_program] with the execution results of our model output program.
For example:
```
Problem:
"Problem: Find the norm and argument (phase angle in radians) of $(1+i) \sqrt{10}$."
```
The gt problem is:
```
"import math from sympy import * i = 1j x = (1+i)*math.sqrt(10) norm = math.sqrt(pow(x.real, 2) + pow(x.imag, 2)) arg = math.atan(x.imag/x.real) print('Norm:', norm) print('Arg:', arg) "
```
Our model doesn't know to output the string "Norm", "Arg".
May I ask if you have any plans to release the relevant evaluation code or provide some ideas?
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.