AlibabaResearch / AlibabaResearch/DAMO-ConvAI
Excecution Accuracy Metric definition incorrect (?)
- 主要言語
- Python
- スター
- 1.6k
- フォーク
- 250
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
Hello,
In the evaluation scripts the following piece of code is being used to generate scores for Execution Accuracy
```
def execute_sql(predicted_sql,ground_truth, db_path):
conn = sqlite3.connect(db_path)
# Connect to the database
cursor = conn.cursor()
cursor.execute(predicted_sql)
predicted_res = cursor.fetchall()
cursor.execute(ground_truth)
ground_truth_res = cursor.fetchall()
res = 0
if set(predicted_res) == set(ground_truth_res):
res = 1
return res
```
Given that the retrieved result sets are turned into `sets` isn't this ignoring (a) DISTINCT errors and (b) incorrect row ordering errors.
For (a), given that the ground truth query might include a `DISTINCT` clause but the generated one will not then an EX of 1 is still assigned.
Example:
```
In [1]: res = [('apple',), ('pear',)]
In [2]: gen_res = [('apple',), ('apple',), ('pear',)]
In [3]: set(res) == set(gen_res)
Out[3]: True
```
For (b) given that the ground truth query might include a `ORDER BY` clause but the generated one will not then an EX of 1 is still assigned.
Example:
```
In [4]: res = [('apple',), ('pear',)]
In [5]: gen_res = [('pear',),('apple',)]
In [6]: set(res) == set(gen_res)
Out[6]: True
```
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。