apache / apache/doris

[Bug] External Iceberg columns with mixed-case names are unreadable: "schema mapping is missing projected column"

Open
#66,811 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

master (reproduced on the ~2026-08-16 tip, b9ee837a3e)

### What's Wrong?

After the external-catalog connector refactor (#64304 line), selecting any column whose name contains an uppercase letter from an external Iceberg table fails:

```
ERROR 1105 (HY000): errCode = 2, detailMessage = [INTERNAL_ERROR]schema mapping is missing
projected column 'channelId'; the schema info from FE is inconsistent with the scan projection (file: ...)
```

Plain scalar columns fail the same way as complex ones — any camelCase name is enough. All-lowercase columns read fine.

Root cause, from tracing it in production: `IcebergSchemaUtils.buildCurrentSchema` (fe-connector-iceberg) keys the current-schema (`-1`) dictionary's top-level TField names off the **lowercased** requested names (`IcebergScanPlanProvider.requestedLowerNames` — the column handles are already lowercased). But the BE scan slot names preserve the Iceberg column case, and `ParquetReader::_do_init_reader` (the "schema mapping is missing projected column" guard) compares slot names against the dictionary **verbatim**. So the dict says `channelid`, the slot says `channelId`, and the guard refuses. The doc comment on `requestedLowerNames` asserts the dict names equal the BE scan-slot names "by construction" — that only holds for tables whose columns are already lowercase. The pre-refactor fe-core path built this dictionary from the Doris `Column` names (real case, `ExternalUtil`), which is why the same tables read fine before the decoupling.

### What You Expected?

Column reads work regardless of name case, as they did via the legacy fe-core Iceberg path.

### How to Reproduce?

1. Any external Iceberg table (REST catalog in my case) with a column like `channelId STRING`.
2. `SELECT channelId FROM cat.db.t LIMIT 1;` → the error above. A lowercase sibling column on the same table reads fine.

### Anything Else?

Hit this while rebasing our production build onto master — MongoDB-derived lakehouse tables are almost entirely camelCase, so every one of them was unreadable. We're running a fix in production: emit the schema's own field case for the dictionary's top-level names and keep the lowered name only as the `caseInsensitiveFindField` lookup key (the equality-delete carrier can stay lowercased — it's consumed strictly by field id).

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with fe-connector-iceberg's IcebergSchemaUtils.buildCurrentSchema and IcebergScanPlanProvider.requestedLowerNames, then inspect ParquetReader::_do_init_reader and its schema-mapping guard. Reproduce the failure with an external Iceberg table containing a camelCase column and compare dictionary names with scan-slot names. Done means mixed-case and lowercase columns both read successfully, including the reported SELECT.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.