AlibabaResearch / AlibabaResearch/DAMO-ConvAI
Excecution Accuracy Metric definition incorrect (?)
- Lenguaje dominante
- Python
- Estrellas
- 1.6k
- Forks
- 250
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
Hello,
In the evaluation scripts the following piece of code is being used to generate scores for Execution Accuracy
```
def execute_sql(predicted_sql,ground_truth, db_path):
conn = sqlite3.connect(db_path)
# Connect to the database
cursor = conn.cursor()
cursor.execute(predicted_sql)
predicted_res = cursor.fetchall()
cursor.execute(ground_truth)
ground_truth_res = cursor.fetchall()
res = 0
if set(predicted_res) == set(ground_truth_res):
res = 1
return res
```
Given that the retrieved result sets are turned into `sets` isn't this ignoring (a) DISTINCT errors and (b) incorrect row ordering errors.
For (a), given that the ground truth query might include a `DISTINCT` clause but the generated one will not then an EX of 1 is still assigned.
Example:
```
In [1]: res = [('apple',), ('pear',)]
In [2]: gen_res = [('apple',), ('apple',), ('pear',)]
In [3]: set(res) == set(gen_res)
Out[3]: True
```
For (b) given that the ground truth query might include a `ORDER BY` clause but the generated one will not then an EX of 1 is still assigned.
Example:
```
In [4]: res = [('apple',), ('pear',)]
In [5]: gen_res = [('pear',),('apple',)]
In [6]: set(res) == set(gen_res)
Out[6]: True
```
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.