apache / apache/incubator-xtable

Delta to Iceberg conversion loses the field ids of map and list children under column mapping

Open
#911 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Search before asking

I searched the open and closed issues; nothing covers this.

### Describe the problem

Converting a Delta table that has column mapping enabled and contains a `map` or `array` column produces an Iceberg table that can't be read, once the Delta writer is 3.x. Reading it fails in Iceberg's Parquet reader:

```
java.lang.IllegalArgumentException: [col-127c1280-..., key_value, key] required binary key (STRING) is not in the store: [...]
```

`ITConversionController#testColumnMappingEnabledDeltaToIceberg` covers this path and passes on Delta 2.4, which the build still uses. It fails against Delta 3.3, which is what surfaced this.

### Root cause

Delta 3.x writes Parquet field IDs; Delta 2.4 doesn't. The difference is visible in the footer of a file written by the same code against each version, for a table with `delta.columnMapping.mode = name`:

```
delta-core 2.4.0 optional group col-0284b2db-... (MAP) { repeated group key_value { ... } }
delta-spark 3.3.3 optional group col-d660aa27-... (MAP) = 2 { repeated group key_value { ... } }
```

XTable's Delta to Iceberg path relies on Iceberg's name mapping, which `IcebergConversionTarget#createAndSetNameMapping` populates with Delta's physical names. Iceberg applies a name mapping only to files that carry no field IDs at all. Delta 3.x's IDs therefore switch Iceberg to ID-based resolution, and Delta assigns column mapping IDs to struct fields only, never to a map key, a map value, or a list element. Those nested fields end up with no ID in the file and no name mapping to fall back on, so they can't be resolved. A map key is required, so the read fails rather than returning nulls.

Delta's own answer is `delta.enableIcebergCompatV2`, which hooks `DeltaParquetWriteSupport` specifically to, in its words, "write nested field IDs for list and map types to the parquet schema". With that enabled the IDs are in the file, and Delta records them per field under `delta.columnMapping.nested.ids`, keyed by the path the child takes in the file, for example `col-1234.key`.

### What the protocol says

`PROTOCOL.md` defines `delta.columnMapping.nested.ids` under "Writer Requirements for IcebergCompatV2", so Delta writes it only when `delta.enableIcebergCompatV2` is set. Under plain column mapping, in `mode=id` and `mode=name` alike, a list's `element` and a map's `key` and `value` have no identity mechanism at all, and those Parquet nodes carry no `field_id`.

Writing `field_id` for every `StructField` is a general column mapping requirement rather than a mode-gated one; the modes differ only in how a reader resolves the IDs. That matches the footer diff above: the 3.x file has IDs on the struct fields, and Iceberg switches to ID-based resolution as soon as any ID is present.

There is no `delta.columnMapping.nested.physicalNames` key. A collection node's physical path is its nearest ancestor `StructField`'s `physicalName` plus `.element`, `.key`, or `.value`.

### What this is not

Ruled out by experiment:

- Iceberg 1.10.2 in place of 1.9.2.
- Parquet 1.13.1 in place of 1.15.2.
- An ID or name bug on the XTable side. The Delta schema JSON and the ID-to-physical-name map that `IcebergSchemaExtractor` builds are identical between Delta 2.4 and 3.3.

### Suggested fix

The two cases need different answers, and only the first is a carrying problem:

1. **IcebergCompatV2 enabled.** The IDs exist in the file. `DeltaSchemaExtractor` reads `delta.columnMapping.nested.ids` and puts those IDs on the converted map key, map value, and list element fields. `IcebergSchemaExtractor` already prefers a field's own ID over one it generates, so the Iceberg schema then matches the file. #912 covers this.

2. **IcebergCompatV2 disabled.** There is nothing to carry. Any ID in the generated Iceberg schema for a collection node would have to be invented, which is the condition that produces the unreadable table. On Delta 3.x such a table isn't convertible to a readable Iceberg table when it contains collections, so the open question is whether XTable refuses the sync, warns at sync time, or documents the limit.

The Spark 3.5 step carries a characterization test for the current behavior, `ITConversionController#testColumnMappingWithoutIcebergCompatIsNotReadableAsIceberg`. It builds a column mapped table without IcebergCompatV2, syncs it to Iceberg, asserts the sync reports SUCCESS, and asserts that reading the resulting Iceberg table throws. Refusing the sync has the advantage that the failure is otherwise silent until someone reads a collection column, which is what that test pins. If the project prefers to refuse or warn, that test is the place that changes.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

*This issue was created with AI assistance.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ITConversionController#testColumnMappingWithoutIcebergCompatIsNotReadableAsIceberg and the existing DeltaSchemaExtractor and IcebergSchemaExtractor paths. Confirm the current behavior for collection fields without IcebergCompatV2, then follow the project’s chosen policy—refuse or warn during sync—and update the characterization test to verify that outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.