REST: lazy-load unreferenced snapshots when snapshot-loading-mode=refs
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 121
Description
### Proposed Change
When `snapshot-loading-mode=refs` is set on the REST catalog, `LoadTable` sends `?snapshots=refs` and the server returns only snapshots referenced by a named branch or tag. This reduces response size for tables with long snapshot histories.
However, there is currently no way for a caller to lazily fetch the remaining snapshots if they are needed — for example, to time-travel to a snapshot that is not referenced by any branch or tag. Such a request surfaces as a plain "snapshot not found" error with no signal that the snapshot may exist but was simply not loaded.
The Java `RESTSessionCatalog` handles this via a `snapshotsSupplier` — a lazy supplier that fetches the full snapshot list on demand when a snapshot lookup fails. iceberg-go has no equivalent.
**Proposed change:**
- When a snapshot lookup fails and `snapshot-loading-mode=refs` is in use, the catalog should detect that the metadata may be incomplete and trigger a follow-up `loadTable` call with `?snapshots=all` to obtain the full snapshot list.
- Alternatively, expose a mechanism for callers to explicitly request a full reload.
- The error returned to the caller should distinguish "snapshot absent from partial metadata" from "snapshot genuinely does not exist".
**Spec reference:**
REST Catalog Open API – loadTable:
https://redocly.github.io/redoc/?url=https://raw.githubusercontent.com/apache/iceberg/apache-iceberg-1.10.1/open-api/rest-catalog-open-api.yaml#tag/Catalog-API/operation/loadTable
**Related:**
- #1748 — added `snapshot-loading-mode` catalog property
- apache/iceberg-go#1749 — implementation PR; lazy loading deferred as follow-up
**Willingness to contribute**
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
### Proposal document
_No response_
### Specifications
- [x] Table
- [ ] View
- [x] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other
Contributor guide
Research direction
Start at the REST catalog's LoadTable path and the snapshot-loading-mode implementation from #1748; compare its behavior with Java's RESTSessionCatalog snapshotsSupplier. Done means a refs load can obtain snapshots=all when an unreferenced snapshot is needed, while genuinely absent snapshots produce a distinct error, with REST behavior covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100