apache / apache/iceberg-python
Pluggable Scan Planning
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Feature Request / Improvement
Umbrella issue. Today we have tightly coupled query planning in Python. This works well but runs into limitations when it comes to concurrency and performance. Having pluggable query planning will allow for:
- **Python based query planning** which is working today and is proven.
- **Iceberg-Rust based query planning** as suggeted in https://github.com/apache/iceberg-python/issues/2159#issuecomment-3033339213.
- **REST Catalog server side planning** this leverages the REST Catalog to plan the query, rather than the client. First issue can be found here: https://github.com/apache/iceberg-python/issues/2302.
As an API, I think it makes sense to follow the Server Side planning. So a `PlanTableScanRequest` as an input, and the [ScanTasks](https://github.com/apache/iceberg/blob/68a241615e104d97a86c932c381d7dfac802d769/open-api/rest-catalog-open-api.py#L1290) as the output.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.