apache / apache/incubator-xtable

Support for deletion vector translation

Open
#339 5 comments 8 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

Deletion vectors is an optimization feature that can be enabled on Delta Lake tables and Iceberg tables. They allow DELETE and UPDATE operations to mark existing rows as removed or changed without rewriting the Parquet file. Hudi may soon support a similar representation for deletion vectors.

Currently, XTable does not support handling and translating the deletion files between formats. This means that XTable cannot preserve the deletion vectors when converting a table from one format to another, resulting in incomplete translation and/or incorrect results. This feature request is to add support for deletion vector translation in XTable.

The proposed steps to implement the first phase of this feature are:

- [x] #340
- [ ] #341
- [ ] #342
- [ ] #343
- [ ] #344
- [ ] #346
- [ ] #345
- [ ] #347
- [ ] #348

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the completed checklist item #340 and the remaining subtasks #341-#348, then trace XTable's existing translation paths for Delta Lake and Iceberg tables. Done means deletion files are handled and preserved when converting between supported formats without incomplete or incorrect results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.