apache / apache/incubator-xtable
Support for deletion vector translation
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
Deletion vectors is an optimization feature that can be enabled on Delta Lake tables and Iceberg tables. They allow DELETE and UPDATE operations to mark existing rows as removed or changed without rewriting the Parquet file. Hudi may soon support a similar representation for deletion vectors.
Currently, XTable does not support handling and translating the deletion files between formats. This means that XTable cannot preserve the deletion vectors when converting a table from one format to another, resulting in incomplete translation and/or incorrect results. This feature request is to add support for deletion vector translation in XTable.
The proposed steps to implement the first phase of this feature are:
- [x] #340
- [ ] #341
- [ ] #342
- [ ] #343
- [ ] #344
- [ ] #346
- [ ] #345
- [ ] #347
- [ ] #348
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the completed checklist item #340 and the remaining subtasks #341-#348, then trace XTable's existing translation paths for Delta Lake and Iceberg tables. Done means deletion files are handled and preserved when converting between supported formats without incomplete or incorrect results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100