apache / apache/iceberg-rust

Feature request: list namespaces and tables in stream style

Open
#1,291 4 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

Hi team, I plan to use `list_namespaces` and `list_tables` on recovery.

For example, when a server goes back, created catalog calls list operations to
- Get what tables to recover;
- For each table, recover states before crash by walking through version hint, metadata, etc to data files.

One feature I would like to have is to stream-style list operation, so recovery process (which is also async via tokio) could follow through.

Current interface only supports to get all entities in one time.
https://github.com/apache/iceberg-rust/blob/b4bc6dd15ae4f7a4b1c9ce555e3458089a83e228/crates/iceberg/src/catalog/mod.rs#L42-L44

https://github.com/apache/iceberg-rust/blob/b4bc6dd15ae4f7a4b1c9ce555e3458089a83e228/crates/iceberg/src/catalog/mod.rs#L73-L74

Related issue:
- Support pagination for list operations: https://github.com/apache/iceberg-rust/issues/1251
+ I would like to spend some time on it, which should unblock my work

### Describe the solution you'd like

Opendal supports stream-style list, if I were to implement the feature, I would very likely mimic the same interface: https://github.com/apache/opendal/blob/faf12983a0f46a5c75641d1a0b0eaa5d0600b972/core/src/types/operator/operator.rs#L1337-L1371

### Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

Contributor guide

Open the contributing guide

Research direction

Start with the list_namespaces and list_tables interfaces in crates/iceberg/src/catalog/mod.rs, then compare the stream-style interface in OpenDAL's operator.rs. Review related issue #1251 before deciding the API shape; done means both catalog list operations can be consumed as streams for recovery.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.