apache / apache/hudi

fix(spark): extended SQL parser enforces ANSI reserved keywords on Spark 4.x, rejecting TIMESTAMP columns in BLOB/VECTOR DDL

Open Beginner friendly
#19,449 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

All six per-version extended SQL parsers set

```scala
parser.SQL_standard_keyword_behavior = conf.ansiEnabled
```

(`HoodieSpark3_3ExtendedSqlParser.scala:88` through `HoodieSpark4_2ExtendedSqlParser.scala:90`), but Spark's own parser wires this flag to `conf.enforceReservedKeywords` (`spark.sql.ansi.enforceReservedKeywords`), which defaults to **false** even when ANSI mode is on.

Spark 4.x defaults `spark.sql.ansi.enabled=true`, so the forked parser enforces reserved keywords that stock Spark leaves off. Any statement routed to the extended parser (anything containing a ` blob` or ` vector` token per `isHoodieCommand`) then rejects SQL the stock parser accepts:

```sql
CREATE TABLE t (id BIGINT, ts TIMESTAMP, data BLOB) USING hudi
-- Spark 4.x: ParseException: no viable alternative at input 'TIMESTAMP'
```

The same statement without the BLOB column parses fine. Blast radius includes a bare `TIMESTAMP` column type, `TIMESTAMP '...'` literals, and ANSI-reserved column names (`user`, `left`, `filter`, `some`, `current_date`).

**Fix**

One line in each of the six parsers: set the flag from `conf.enforceReservedKeywords` instead of `conf.ansiEnabled`, matching Spark's `AbstractParser`. The conf exists since Spark 3.3.0, so it is safe on all supported profiles.

The 3.5 and regenerated 4.1 forked grammars behave identically once the flag matches -- the grammar itself has no gap.

Found while reviewing #19408; its tests use `DATE` instead of `TIMESTAMP` to work around this.

**Environment Description**

* Hudi version: master
* Spark version: 4.0/4.1/4.2 profiles (any profile with `spark.sql.ansi.enabled=true`)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the six per-version extended SQL parser files, from HoodieSpark3_3ExtendedSqlParser.scala:88 through HoodieSpark4_2ExtendedSqlParser.scala:90, and compare their reserved-keyword setting with Spark's AbstractParser. Review the DATE-based tests mentioned from #19408, then verify that the reported TIMESTAMP and BLOB DDL parses with reserved-keyword enforcement controlled by the dedicated configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.