apache / apache/iceberg-rust

[RestCatalog] Support Page Sizes

Open
#2,934 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

By [the REST spec](https://iceberg.apache.org/docs/latest/catalog-properties/#rest-catalog-properties), a REST catalog may return the `rest-page-size` property in a `/v1/config` response. The catalog client is then supposed to use it and provide it along with list requests via the `?pageSize` query parameter.

Today, the Rust `RestCatalog` client doesn't look at the `rest-page-size` property and never sends a `pageSize` query parameter. It therefore depends on the catalog server to implement an implicit default page size.

### Describe the solution you'd like

The spec already prescribes this fairly precisely.

We'd use a precedence order similar to other properties:

1. catalog default from `/v1/config`: `{"defaults": {"rest-page-size": "1000"}}` (if provided)
2. overridden by client side config: `with_page_size(1000)` (else `None` or some client-side default[^1])
3. overridden by catalog overrides from `v1/config`: `{"overrides": {"rest-page-size": "1000"}}` (if provided)

[^1]: IIUC only `iceberg-go` currently sets a client-side default (20). Java and Python don't.

### Willingness to contribute

I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start with the Rust RestCatalog handling for /v1/config and list requests. Trace how catalog defaults, with_page_size(1000), and catalog overrides should be combined, then verify that list requests send the resulting value as the pageSize query parameter; done means the REST spec precedence is respected.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.