apache / apache/datafusion

Regression: SQL parser became less tolerant to columns that match SQL keywords

Open
#17,802 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Latest release (between `49.0.2` and `50.0.0`) seems to introduce a regression in SQL parser that causes it to be less tolerant of column names that collide with SQL keywords like `limit` and `offset`.

### To Reproduce

Using `datafusion-cli`:
```sql
> create or replace table x as (
select 1 as offset
);

> select * from x;
+--------+
| offset |
+--------+
| 1 |
+--------+

-- NOTE: Disambiguates correctly
> select offset from x;
+--------+
| offset |
+--------+
| 1 |
+--------+

-- NOTE: Adding a literal in the front breaks disambiguation
> select 'a' as a, offset from x;
🤔 Invalid statement: SQL error: ParserError("Expected: end of statement, found: x at Line: 1, Column: 30")

-- NOTE: Adding a literal in the back - does not!
> select offset, 'a' as a from x;
+--------+---+
| offset | a |
+--------+---+
| 1 | a |
+--------+---+
1 row(s) fetched.

-- NOTE: Quoting helps, but is unfortunate
> select 'a' as a, "offset" from x;
+---+--------+
| a | offset |
+---+--------+
| a | 1 |
+---+--------+
1 row(s) fetched.
```

In Datafusion `49.0.2` disambiguation works fine in all cases, so it's a regression.

Same behaviour reproduces for other keywords like `limit`.

### Expected behavior

SQL and its many dialects are rich with keywords, so context-aware parsing really helps to keep queries free of excessive quoting.

It so happens that `offset` column is present in *hundreds* of our datasets, so if this behavior change is highly impactful and I hope can be reversed.

### Additional context

The cause might be in `sqlparser` crate (between versions `0.55.0` and `0.58.0`). Will attempt to isolate the issue there and create a linked ticket.

Contributor guide

Open the contributing guide

Research direction

Reproduce the queries in datafusion-cli, especially the literal-before-column cases using offset and limit, and compare the behavior with DataFusion 49.0.2. Investigate the possible sqlparser crate change between versions 0.55.0 and 0.58.0; done means keyword-named columns parse correctly without requiring quotes, including when a literal precedes them.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.