apache / apache/incubator-xtable
Delta Kernel target writes column-mapping IDs with column mapping disabled
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
Translating an Iceberg schema with field IDs to Delta through the Kernel target produces a table that delta-rs 1.6.3 cannot open:
```
Kernel error: Schema error: Column mapping is not enabled but field 'id' is annotated with delta.columnMapping.id
```
`DeltaKernelSchemaExtractor.getFieldMetadata()` copies `InternalField.fieldId` into `delta.columnMapping.id`. The target does not enable `delta.columnMapping.mode`, and the resulting protocol remains reader 1 / writer 2. The same emission is present on current upstream `main`.
To reproduce:
1. Sync an Iceberg snapshot whose schema contains a `long` field named `id` with field ID 1 to a fresh Delta Kernel target.
2. Inspect the first Delta commit: `schemaString` contains `"metadata":{"delta.columnMapping.id":1}`, configuration has no `delta.columnMapping.mode`, and the protocol is `{"minReaderVersion":1,"minWriterVersion":2}`.
3. Open the output using Python `deltalake` 1.6.3: `DeltaTable(path)`. It fails with the error above before reading data.
Expected: omit column-mapping annotations when the target does not enable column mapping. Keep reader 1 / writer 2 for these tables. Source-side extraction of IDs from column-mapped Delta tables should continue to work.
Locally, removing the five-line ID propagation block lets delta-rs 1.6.3 open the translated table and read both rows from a synthetic Parquet fixture, with the protocol unchanged. Regression checks cover nested structs, arrays, maps, preserved UUID/comment metadata, and the committed protocol. The current Spark target already uses `SparkSchemaExtractor`, which does not emit these annotations.
Reference: [Delta column-mapping protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#column-mapping).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at DeltaKernelSchemaExtractor.getFieldMetadata() and inspect the Delta target’s first commit generation, then reproduce the schema with an Iceberg field ID. Run the regression checks, including nested structs, arrays, maps, UUID/comment metadata, and protocol assertions; done means delta-rs 1.6.3 opens the output while reader 1 / writer 2 remains unchanged and source-side ID extraction still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100