apache / apache/iceberg-python

Add support for bucket expression to table scans

Open
#3,839 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
78

Description

### Feature Request / Improvement

For time partitioning, we can express time range expressions and they lead to partition pruning when planning an Arrow scan:

```py
scan = table.scan(
row_filter=And(
GreaterThanOrEqual("event_ts", start),
LessThan("event_ts", end),
)
)

arrow_table = scan.to_arrow()
```

It would be useful to be able to filter by other hidden partitioning transforms, such as buckets -- e.g. filtering on `bucket[16](user_id) in {0, 1, 2, 3}`, and getting partition pruning whenever possible based on the underlying table partitioning specs (falling back to filtering rows when files cannot be pruned).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the table.scan and scan.to_arrow entry points described in the issue, then trace the existing time-range expression handling used for Arrow partition pruning. Check how partition specs and row filters are applied; done means bucket expressions such as bucket[16](user_id) can prune files when possible and fall back to row filtering when they cannot.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.