apache / apache/incubator-xtable

Delta Kernel target writes column-mapping IDs with column mapping disabled

Open
#936 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

Translating an Iceberg schema with field IDs to Delta through the Kernel target produces a table that delta-rs 1.6.3 cannot open:

```
Kernel error: Schema error: Column mapping is not enabled but field 'id' is annotated with delta.columnMapping.id
```

`DeltaKernelSchemaExtractor.getFieldMetadata()` copies `InternalField.fieldId` into `delta.columnMapping.id`. The target does not enable `delta.columnMapping.mode`, and the resulting protocol remains reader 1 / writer 2. The same emission is present on current upstream `main`.

To reproduce:

1. Sync an Iceberg snapshot whose schema contains a `long` field named `id` with field ID 1 to a fresh Delta Kernel target.
2. Inspect the first Delta commit: `schemaString` contains `"metadata":{"delta.columnMapping.id":1}`, configuration has no `delta.columnMapping.mode`, and the protocol is `{"minReaderVersion":1,"minWriterVersion":2}`.
3. Open the output using Python `deltalake` 1.6.3: `DeltaTable(path)`. It fails with the error above before reading data.

Expected: omit column-mapping annotations when the target does not enable column mapping. Keep reader 1 / writer 2 for these tables. Source-side extraction of IDs from column-mapped Delta tables should continue to work.

Locally, removing the five-line ID propagation block lets delta-rs 1.6.3 open the translated table and read both rows from a synthetic Parquet fixture, with the protocol unchanged. Regression checks cover nested structs, arrays, maps, preserved UUID/comment metadata, and the committed protocol. The current Spark target already uses `SparkSchemaExtractor`, which does not emit these annotations.

Reference: [Delta column-mapping protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#column-mapping).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at DeltaKernelSchemaExtractor.getFieldMetadata() and inspect the Delta target’s first commit generation, then reproduce the schema with an Iceberg field ID. Run the regression checks, including nested structs, arrays, maps, UUID/comment metadata, and protocol assertions; done means delta-rs 1.6.3 opens the output while reader 1 / writer 2 remains unchanged and source-side ID extraction still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.