apache / apache/iceberg-python

Add Server-Side Scan Planning Support

未关闭
#2,775 5 条评论 5 个 reaction 已指派 1 人 已被 @geruh 认领 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

## Feature Request / Improvement

Now that Java has added server-side scan planning support [PR #14480](https://github.com/apache/iceberg/pull/14480), I believe Python is a great place to integrate this functionality! We have all the building blocks and they are almost brought to completion. I'm creating this issue to track all of the tasks we need to drive it through

## Context

We have some open PRs with some needed model changes, but we have pivoted to using our existing models and ensuring they're properly serializable with pydantic.

For example, initially we can work on:

1. Expression Serialization to ensure `BooleanExpression` and subclasses serialize correctly for REST API. Related to @Fokko's work on #2750 and @rambleraptor #2435.

2. DataFile Serialization to ensure we can properly deserialize the data/deletefiles from the server response. Open API uses kebab-case (`file-format`, `file-path`) but our models expect snake_case (`file_format`, `file_path`).

## REST API Endpoints to Implement

Based on the [Iceberg REST spec](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml):

1. `POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/plan` - Submit scan for planning
2. `GET /v1/{prefix}/namespaces/{namespace}/tables/{table}/plan/{plan-id}` - Fetch planning result
3. `DELETE /v1/{prefix}/namespaces/{namespace}/tables/{table}/plan/{plan-id}` - Cancel planning
4. `POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/tasks` - Fetch scan tasks for a plan task

## Tasks

Initially we can start with core sync planning and once that's in place we can add the async support as it looks like it exists in https://github.com/apache/iceberg/blob/main/core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java

### Core Sync Planning

- [x] Build on @Fokko's expression work to ensure `Expression` classes serialize properly with Pydantic.
- #2783
- [x] Construct only plan Request/Response models for synchronous planning
- [x] Add support for REST scan to fetch scan tasks replicating `DataScan` behavior
- [x] Parse server response to `FileScanTask` objects (handle Data/DeleteFile construction)
- [x] Add `plan_table_scan()` methods
- [ ] Add documentation

### Full Scan planning support (Follow-up)

Complete the full scan planning API with async operations and pagination.
- [ ] Add the rest of the models for async planning
- [ ] Add support for endpoint 2 & 3 to RESTScan
- [ ] Add endpoint 4 support to `RestCatalog`
- [ ] Complete documentation with all scenarios

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。