apache / apache/iceberg-python
Support data files compaction
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
Introduce an API to compact data files. The first version of the API will do the following:
- take a predicate expression as input parameter to find data files matching the filter that will be re-written
- group data files by partitions and rewrite them using the same bin-packing constraints of the writer
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading PyIceberg's existing data-file and writer APIs to understand how predicate filtering, partition grouping, and bin-packing constraints are represented. Trace the writer path and relevant tests before deciding where the compaction API belongs. Done means callers can provide a predicate, matching files are grouped by partition, and rewritten files use the writer's existing bin-packing constraints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100