apache / apache/iceberg

feat: Delta to Iceberg v3 conversion support tracking issue

Open
#17,146 0 comments 0 reactions 1 assignee Claimed by @vladislav-sidorovich View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

Tracking issue for updating `delta-lake` module to support conversion from Delta Lake format to Iceberg v3 format.

Initial update PR: https://github.com/apache/iceberg/pull/15407

## Millestones:
- [x] Converts all data types
- [x] Converts table schema and partitions spec
- [x] Support UPDATES and DELETS (Delta Lake Remove action)
- [x] Support Delta VACUUM scenario
- [x] Edge case when no check point files exist
- [x] Support DVs
- [ ] Handle Schema evolution
- [ ] Handle Edge cases for partitions and Generated columns
- [ ] Support catalog-managed tables (Delta 4.1.0 +)
- [ ] Support All Delta Lake actions
- [ ] Support All Delta Lake features (column mapping, generated columns and so on)
- [ ] Incremental Conversion (from/to a specific Delta Version)

## Detailed plan
### SCHEMA & METADATA

- [x] Map Delta (Spark) primitive types (IntegerType, StringType, TimestampNTZType) to Iceberg primitive types.
- [x] Map complex types (StructType, ArrayType, MapType) recursively.
- [ ] Implement ID assignment strategy (map Delta's physical IDs if columnMapping is enabled, otherwise generate sequential IDs).
- [x] Translate column nullability and documentation/comments.

### TABLE LAYOUT & PARTITIONING

- [ ] Parse Delta's partitionColumns from the metadata action.
- [ ] Analyze Delta's generatedColumns expressions (e.g., CAST(ts AS DATE), YEAR(ts)).
- [ ] Map Delta generated expressions to Iceberg Transforms (day(ts), year(ts)).
- [ ] Map explicit (non-generated) partition columns to Iceberg 'identity' transforms.
- [ ] Read Delta's latest OPTIMIZE commit info to extract zOrderBy or Liquid Clustering configurations.
- [ ] Translate Delta OPTIMIZE into an Iceberg SortOrder attached to the table metadata.

### DATA FILES & STATISTICS TRANSLATION

- [x] Handle Add/Remove Delta Actions
- [ ] Extract minValues, maxValues, nullCount, and numRecords from Delta metadata (add action)
- [ ] Convert Delta string stats to Iceberg's truncated byte buffers (handle 16-byte truncation and upper-bound increments)
- [ ] Handle the NaN gap (leave Iceberg nan_value_counts empty).
- [x] Handle VACUUM operation for Delta Lake tables

### HANDLE ALL DELTA LAKE PROTOCOL ACTION

- [ ] Metadata (Schema definition/evolution) action
- [x] Add action
- [x] Remove action
- [ ] CDC action
- [ ] Txn action
- [ ] Protocol action
- [ ] Commit info action
- [ ] Domain metadata action
- [ ] Sidecar action
- [ ] Check point action
- [ ] Reconciliation action

### ADVANCED FEATURES & MUTABILITY

- [x] Deletion Vectors Support
- [ ] Table Features & Properties
- [ ] Catalog-managed tables https://delta.io/blog/2026-02-02-delta-catalog-managed-tables/

### CONTINUOUS SYNC

- [ ] Store the last synced Delta version in the Iceberg table properties (e.g., delta_sync_version=15).
- [ ] Build a poller that reads Delta commits starting from delta_sync_version + 1.
- [ ] Handle on-the-fly schema evolution (apply Iceberg UpdateSchema API if a new Delta commit contains altered Metadata).

### CLEAN-UP

- [ ] Remove usage of Delta Kernel internal classes

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.