apache / apache/iceberg-python

Support data files compaction

Open
#1,092 13 comments 33 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
78

Description

Introduce an API to compact data files. The first version of the API will do the following:
- take a predicate expression as input parameter to find data files matching the filter that will be re-written
- group data files by partitions and rewrite them using the same bin-packing constraints of the writer

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading PyIceberg's existing data-file and writer APIs to understand how predicate filtering, partition grouping, and bin-packing constraints are represented. Trace the writer path and relevant tests before deciding where the compaction API belongs. Done means callers can provide a predicate, matching files are grouped by partition, and rewritten files use the writer's existing bin-packing constraints.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.