apache / apache/datafusion-comet

Iceberg native scan fails queries on tables whose column names are case-distinct to Java but not to Rust

Open
#5,540 1 comment 0 reactions 0 assignees View on GitHub
area:scan bug priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

The native Iceberg scan builds its schema adapter with no JVM case tables and case sensitivity hardcoded to false (iceberg_scan.rs:226), so case-insensitive name matching falls back to Rust's str::to_lowercase. Rust ships newer Unicode data than the JDK, so codepoints that gained lowercase mappings in Unicode 14+ fold together in Rust while Java keeps them distinct. A table with two such columns is perfectly legal to Spark, but Comet's adapter folds them into a duplicate and the query dies with _LEGACY_ERROR_TEMP_2093 Found duplicate field(s) ... in case-insensitive mode, an error Spark itself would never raise.

The same divergence class was fixed for the parquet and Delta scan paths in #5365 by shipping the running JVM's case data to native (JvmCaseTables); the Iceberg path is the remaining consumer of the fallback (there's a code comment marking it).

### Steps to reproduce

Columns are U+10570 (VITHKUQI CAPITAL LETTER A) and U+10597 (its lowercase), a Unicode-14 case pair that JDK 17 treats as distinct. They may render as boxes depending on your font; copy-paste still works, or use the constructed variant below.

```sql
CREATE TABLE hadoop_catalog.uni_case (`U+10570` INT, `U+10597` INT) USING iceberg;
INSERT INTO hadoop_catalog.uni_case VALUES (1, 2);
SELECT `U+10570`, `U+10597` FROM hadoop_catalog.db.uni_case;
```

Font-independent version (spark-shell):

```scala
val upper = new String(Character.toChars(0x10570))
val lower = new String(Character.toChars(0x10597))
spark.sql(s"CREATE TABLE hadoop_catalog.uni_case (`$upper` INT, `$lower` INT) USING iceberg")
spark.sql(s"INSERT INTO hadoop_catalog.uni_case VALUES (1, 2)")
spark.sql(s"SELECT `$upper`, `$lower` FROM hadoop_catalog.db.uni_case").collect()
```

### Expected behavior

The query returns [1,2] under Comet exactly as under stock Spark. The adapter should also honor spark.sql.caseSensitive rather than hardcoding insensitive matching.

### Additional context

Fix direction: thread the same case tables NativeScanCommon carries through the Iceberg scan config, and pass the real case-sensitivity flag.

Contributor guide

Open the contributing guide

Research direction

Start at iceberg_scan.rs:226 and compare the Iceberg scan configuration with the parquet and Delta paths fixed in #5365. Trace how NativeScanCommon carries JvmCaseTables and how the case-sensitivity flag is configured. Reproduce the U+10570/U+10597 query, then verify that both case-sensitive behavior and stock Spark's case-insensitive result work without duplicate-field errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, rust, scala
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.