apache / apache/datafusion-comet
Iceberg native scan fails queries on tables whose column names are case-distinct to Java but not to Rust
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
The native Iceberg scan builds its schema adapter with no JVM case tables and case sensitivity hardcoded to false (iceberg_scan.rs:226), so case-insensitive name matching falls back to Rust's str::to_lowercase. Rust ships newer Unicode data than the JDK, so codepoints that gained lowercase mappings in Unicode 14+ fold together in Rust while Java keeps them distinct. A table with two such columns is perfectly legal to Spark, but Comet's adapter folds them into a duplicate and the query dies with _LEGACY_ERROR_TEMP_2093 Found duplicate field(s) ... in case-insensitive mode, an error Spark itself would never raise.
The same divergence class was fixed for the parquet and Delta scan paths in #5365 by shipping the running JVM's case data to native (JvmCaseTables); the Iceberg path is the remaining consumer of the fallback (there's a code comment marking it).
### Steps to reproduce
Columns are U+10570 (VITHKUQI CAPITAL LETTER A) and U+10597 (its lowercase), a Unicode-14 case pair that JDK 17 treats as distinct. They may render as boxes depending on your font; copy-paste still works, or use the constructed variant below.
```sql
CREATE TABLE hadoop_catalog.uni_case (`U+10570` INT, `U+10597` INT) USING iceberg;
INSERT INTO hadoop_catalog.uni_case VALUES (1, 2);
SELECT `U+10570`, `U+10597` FROM hadoop_catalog.db.uni_case;
```
Font-independent version (spark-shell):
```scala
val upper = new String(Character.toChars(0x10570))
val lower = new String(Character.toChars(0x10597))
spark.sql(s"CREATE TABLE hadoop_catalog.uni_case (`$upper` INT, `$lower` INT) USING iceberg")
spark.sql(s"INSERT INTO hadoop_catalog.uni_case VALUES (1, 2)")
spark.sql(s"SELECT `$upper`, `$lower` FROM hadoop_catalog.db.uni_case").collect()
```
### Expected behavior
The query returns [1,2] under Comet exactly as under stock Spark. The adapter should also honor spark.sql.caseSensitive rather than hardcoding insensitive matching.
### Additional context
Fix direction: thread the same case tables NativeScanCommon carries through the Iceberg scan config, and pass the real case-sensitivity flag.
Contributor guide
Research direction
Start at iceberg_scan.rs:226 and compare the Iceberg scan configuration with the parquet and Delta paths fixed in #5365. Trace how NativeScanCommon carries JvmCaseTables and how the case-sensitivity flag is configured. Reproduce the U+10570/U+10597 query, then verify that both case-sensitive behavior and stock Spark's case-insensitive result work without duplicate-field errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, rust, scala
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100