AlibabaResearch / AlibabaResearch/DAMO-ConvAI
Excecution Accuracy Metric definition incorrect (?)
- Ngôn ngữ chính
- Python
- Star
- 1.6k
- Fork
- 250
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
Hello,
In the evaluation scripts the following piece of code is being used to generate scores for Execution Accuracy
```
def execute_sql(predicted_sql,ground_truth, db_path):
conn = sqlite3.connect(db_path)
# Connect to the database
cursor = conn.cursor()
cursor.execute(predicted_sql)
predicted_res = cursor.fetchall()
cursor.execute(ground_truth)
ground_truth_res = cursor.fetchall()
res = 0
if set(predicted_res) == set(ground_truth_res):
res = 1
return res
```
Given that the retrieved result sets are turned into `sets` isn't this ignoring (a) DISTINCT errors and (b) incorrect row ordering errors.
For (a), given that the ground truth query might include a `DISTINCT` clause but the generated one will not then an EX of 1 is still assigned.
Example:
```
In [1]: res = [('apple',), ('pear',)]
In [2]: gen_res = [('apple',), ('apple',), ('pear',)]
In [3]: set(res) == set(gen_res)
Out[3]: True
```
For (b) given that the ground truth query might include a `ORDER BY` clause but the generated one will not then an EX of 1 is still assigned.
Example:
```
In [4]: res = [('apple',), ('pear',)]
In [5]: gen_res = [('pear',),('apple',)]
In [6]: set(res) == set(gen_res)
Out[6]: True
```
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.