awslabs / awslabs/unified-text2sql-benchmark

spider test suite eval cannot evaluate fiben

Open
#5 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
88
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Hi, when I use [spider test suite eval](https://github.com/taoyds/test-suite-sql-eval) to evaluate fiben, I got error from `process_sql.py` when passing golden sql, would you like to share you scripts to evaluate fiben?

```
Traceback (most recent call last):
File "/project-path/src/vendor/test-suite-sql-eval/evaluation.py", line 938, in
evaluate(args.gold, args.pred, args.db, args.etype, kmaps, args.plug_value, args.keep_distinct, args.progress_bar_for_each_datapoint)
File "/project-path/src/vendor/test-suite-sql-eval/evaluation.py", line 574, in evaluate
g_sql = get_sql(schema, g_str)
File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 557, in get_sql
_, sql = parse_sql(toks, 0, tables_with_alias, schema)
File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 509, in parse_sql
from_end_idx, table_units, conds, default_tables = parse_from(toks, start_idx, tables_with_alias, schema)
File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 390, in parse_from
idx, table_unit, table_name = parse_table_unit(toks, idx, tables_with_alias, schema)
File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 260, in parse_table_unit
key = tables_with_alias[toks[idx]]
KeyError: '"listedsecurity"'
```

Contributor guide

Open the contributing guide

Research direction

Start with the spider test suite eval command and reproduce the traceback in evaluation.py, tracing the golden SQL through process_sql.py. Check how fiben supplies the golden SQL and schema, especially the "listedsecurity" table lookup. Done means the evaluation path is documented or the reported parsing failure is resolved and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.