Feature request: list namespaces and tables in stream style
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
Hi team, I plan to use `list_namespaces` and `list_tables` on recovery.
For example, when a server goes back, created catalog calls list operations to
- Get what tables to recover;
- For each table, recover states before crash by walking through version hint, metadata, etc to data files.
One feature I would like to have is to stream-style list operation, so recovery process (which is also async via tokio) could follow through.
Current interface only supports to get all entities in one time.
https://github.com/apache/iceberg-rust/blob/b4bc6dd15ae4f7a4b1c9ce555e3458089a83e228/crates/iceberg/src/catalog/mod.rs#L42-L44
https://github.com/apache/iceberg-rust/blob/b4bc6dd15ae4f7a4b1c9ce555e3458089a83e228/crates/iceberg/src/catalog/mod.rs#L73-L74
Related issue:
- Support pagination for list operations: https://github.com/apache/iceberg-rust/issues/1251
+ I would like to spend some time on it, which should unblock my work
### Describe the solution you'd like
Opendal supports stream-style list, if I were to implement the feature, I would very likely mimic the same interface: https://github.com/apache/opendal/blob/faf12983a0f46a5c75641d1a0b0eaa5d0600b972/core/src/types/operator/operator.rs#L1337-L1371
### Willingness to contribute
I would be willing to contribute to this feature with guidance from the Iceberg Rust community
Contributor guide
Research direction
Start with the list_namespaces and list_tables interfaces in crates/iceberg/src/catalog/mod.rs, then compare the stream-style interface in OpenDAL's operator.rs. Review related issue #1251 before deciding the API shape; done means both catalog list operations can be consumed as streams for recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100