AlibabaResearch / AlibabaResearch/DAMO-ConvAI

Excecution Accuracy Metric definition incorrect (?)

Aperta
#177 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
1.6k
Fork
250
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Hello,

In the evaluation scripts the following piece of code is being used to generate scores for Execution Accuracy
```
def execute_sql(predicted_sql,ground_truth, db_path):
conn = sqlite3.connect(db_path)
# Connect to the database
cursor = conn.cursor()
cursor.execute(predicted_sql)
predicted_res = cursor.fetchall()
cursor.execute(ground_truth)
ground_truth_res = cursor.fetchall()
res = 0
if set(predicted_res) == set(ground_truth_res):
res = 1
return res
```
Given that the retrieved result sets are turned into `sets` isn't this ignoring (a) DISTINCT errors and (b) incorrect row ordering errors.
For (a), given that the ground truth query might include a `DISTINCT` clause but the generated one will not then an EX of 1 is still assigned.
Example:
```
In [1]: res = [('apple',), ('pear',)]

In [2]: gen_res = [('apple',), ('apple',), ('pear',)]

In [3]: set(res) == set(gen_res)
Out[3]: True
```
For (b) given that the ground truth query might include a `ORDER BY` clause but the generated one will not then an EX of 1 is still assigned.

Example:
```
In [4]: res = [('apple',), ('pear',)]

In [5]: gen_res = [('pear',),('apple',)]

In [6]: set(res) == set(gen_res)
Out[6]: True
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.