fix(spark): extended SQL parser enforces ANSI reserved keywords on Spark 4.x, rejecting TIMESTAMP columns in BLOB/VECTOR DDL
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
All six per-version extended SQL parsers set
```scala
parser.SQL_standard_keyword_behavior = conf.ansiEnabled
```
(`HoodieSpark3_3ExtendedSqlParser.scala:88` through `HoodieSpark4_2ExtendedSqlParser.scala:90`), but Spark's own parser wires this flag to `conf.enforceReservedKeywords` (`spark.sql.ansi.enforceReservedKeywords`), which defaults to **false** even when ANSI mode is on.
Spark 4.x defaults `spark.sql.ansi.enabled=true`, so the forked parser enforces reserved keywords that stock Spark leaves off. Any statement routed to the extended parser (anything containing a ` blob` or ` vector` token per `isHoodieCommand`) then rejects SQL the stock parser accepts:
```sql
CREATE TABLE t (id BIGINT, ts TIMESTAMP, data BLOB) USING hudi
-- Spark 4.x: ParseException: no viable alternative at input 'TIMESTAMP'
```
The same statement without the BLOB column parses fine. Blast radius includes a bare `TIMESTAMP` column type, `TIMESTAMP '...'` literals, and ANSI-reserved column names (`user`, `left`, `filter`, `some`, `current_date`).
**Fix**
One line in each of the six parsers: set the flag from `conf.enforceReservedKeywords` instead of `conf.ansiEnabled`, matching Spark's `AbstractParser`. The conf exists since Spark 3.3.0, so it is safe on all supported profiles.
The 3.5 and regenerated 4.1 forked grammars behave identically once the flag matches -- the grammar itself has no gap.
Found while reviewing #19408; its tests use `DATE` instead of `TIMESTAMP` to work around this.
**Environment Description**
* Hudi version: master
* Spark version: 4.0/4.1/4.2 profiles (any profile with `spark.sql.ansi.enabled=true`)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the six per-version extended SQL parser files, from HoodieSpark3_3ExtendedSqlParser.scala:88 through HoodieSpark4_2ExtendedSqlParser.scala:90, and compare their reserved-keyword setting with Spark's AbstractParser. Review the DATE-based tests mentioned from #19408, then verify that the reported TIMESTAMP and BLOB DDL parses with reserved-keyword enforcement controlled by the dedicated configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100