allenai / allenai/Lila

How to verify whether the model's output is equal to the ground truth?

Offen
#2 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
TeX
Sterne
90
Forks
11
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Thanks for releasing the dataset of Lila benchmark.

Now, I want to evalutate the math reasoing ability by Lila benchmark.

I have a question that has been bothering me: how can we compare whether the ground truth and the model output are strictly equal?

It is difficult to compare the execution results with [output_answer]. And, It is not accurate to compare the actual execution results of the [output_program] with the execution results of our model output program.

For example:
```
Problem:
"Problem: Find the norm and argument (phase angle in radians) of $(1+i) \sqrt{10}$."
```
The gt problem is:
```
"import math from sympy import * i = 1j x = (1+i)*math.sqrt(10) norm = math.sqrt(pow(x.real, 2) + pow(x.imag, 2)) arg = math.atan(x.imag/x.real) print('Norm:', norm) print('Arg:', arg) "
```

Our model doesn't know to output the string "Norm", "Arg".

May I ask if you have any plans to release the relevant evaluation code or provide some ideas?

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.