awslabs / awslabs/unified-text2sql-benchmark
spider test suite eval cannot evaluate fiben
- Dominant language
- Python
- Stars
- 88
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Hi, when I use [spider test suite eval](https://github.com/taoyds/test-suite-sql-eval) to evaluate fiben, I got error from `process_sql.py` when passing golden sql, would you like to share you scripts to evaluate fiben?
```
Traceback (most recent call last):
File "/project-path/src/vendor/test-suite-sql-eval/evaluation.py", line 938, in
evaluate(args.gold, args.pred, args.db, args.etype, kmaps, args.plug_value, args.keep_distinct, args.progress_bar_for_each_datapoint)
File "/project-path/src/vendor/test-suite-sql-eval/evaluation.py", line 574, in evaluate
g_sql = get_sql(schema, g_str)
File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 557, in get_sql
_, sql = parse_sql(toks, 0, tables_with_alias, schema)
File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 509, in parse_sql
from_end_idx, table_units, conds, default_tables = parse_from(toks, start_idx, tables_with_alias, schema)
File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 390, in parse_from
idx, table_unit, table_name = parse_table_unit(toks, idx, tables_with_alias, schema)
File "/project-path/src/vendor/test-suite-sql-eval/process_sql.py", line 260, in parse_table_unit
key = tables_with_alias[toks[idx]]
KeyError: '"listedsecurity"'
```
Contributor guide
Research direction
Start with the spider test suite eval command and reproduce the traceback in evaluation.py, tracing the golden SQL through process_sql.py. Check how fiben supplies the golden SQL and schema, especially the "listedsecurity" table lookup. Done means the evaluation path is documented or the reported parsing failure is resolved and verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100