REST catalog client loops indefinitely when a server repeats a page token
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
None
### Please describe the bug 🐞
`RESTSessionCatalog` keeps requesting pages until a response has a null `next-page-token`.
If a REST catalog returns a token that the client has already used, the listing never terminates. Each repeated response is also added to the result builder, so memory use and request volume continue growing while the call is blocked.
I reproduced this against Iceberg 1.10.0 with a small HTTP stub that always returns the same token.
The same loop structure is present in 1.11.0 and on `main` in these methods:
- `listTables`
- `listNamespaces`
- `listViews`
Current source: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java
### Measured result
The stub returns this response for every namespace listing request:
```json
{"namespaces":[["demo"]],"next-page-token":"stuck"}
```
Results from the same test:
```text
control, next-page-token: null terminated after 1 request
repeated token 51,043 requests in 5 seconds, still running
```
Every individual HTTP request completes successfully, so the connection and socket timeouts do not bound the overall listing operation.
### Expected behavior
The client should fail with a clear error when pagination does not make progress, rather than making
unlimited requests and accumulating repeated results.
Returning a partial listing would be unsafe because callers could treat missing namespaces, tables,
or views as a complete result.
### REST contract
The REST specification requires the last response to have a null `next-page-token`. A server that repeats a token is therefore violating the contract:
https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L2376
This report is about limiting the effect of that invalid response on the client. It is not claiming that repeated tokens are valid server behavior.
### Catalog implementations checked
I used Codex to check the current implementations and pagination tests for Apache Polaris, Project Nessie,
Lakekeeper, Unity Catalog, and Iceberg's REST fixture. None intentionally reuses a page token:
- Polaris advances its listing position and returns null on the final page.
- Nessie includes its underlying paging token only when more entries exist.
- Lakekeeper advances through storage tokens and returns no token when exhausted.
- Unity Catalog currently consumes its internal pages before returning a complete Iceberg listing.
- Iceberg's REST fixture uses increasing numeric offsets and returns null at the end.
It didn't find a released catalog that produces the repeated-token behavior. A catalog defect or an
intermediary returning a stale response could still trigger it.
### Reproduction outline
1. Start an HTTP server that returns a normal `/v1/config` response.
2. Return one namespace and the same non-null `next-page-token` from every `/v1/namespaces` request.
3. Initialize `RESTCatalog` against the server.
4. Call `listNamespaces()`.
5. Observe that the call does not return and requests continue until the process is stopped.
The healthy control changes only `next-page-token` to null and terminates after one request.
### Possible fix
Remember tokens already used by each listing call and reject a token when it appears again. Tracking
all tokens is necessary because comparing only with the previous token does not detect a server that
alternates between two or more tokens.
A page limit would also bound the loop, but it would impose an arbitrary maximum on valid catalogs.
Before preparing a PR, I would like confirmation that failing on a repeated token is the preferred
behavior.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start in core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java and inspect pagination in listTables, listNamespaces, and listViews. Reproduce the issue with an HTTP stub that returns a repeated non-null next-page-token, then verify that each listing fails clearly on token reuse instead of looping or accumulating repeated results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100