apache / apache/iceberg-go

REST: lazy-load unreferenced snapshots when snapshot-loading-mode=refs

Open
#1,792 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
463
Forks
232
Avg merge
2d 11h
Merged PRs (30d)
121

Description

### Proposed Change

When `snapshot-loading-mode=refs` is set on the REST catalog, `LoadTable` sends `?snapshots=refs` and the server returns only snapshots referenced by a named branch or tag. This reduces response size for tables with long snapshot histories.

However, there is currently no way for a caller to lazily fetch the remaining snapshots if they are needed — for example, to time-travel to a snapshot that is not referenced by any branch or tag. Such a request surfaces as a plain "snapshot not found" error with no signal that the snapshot may exist but was simply not loaded.

The Java `RESTSessionCatalog` handles this via a `snapshotsSupplier` — a lazy supplier that fetches the full snapshot list on demand when a snapshot lookup fails. iceberg-go has no equivalent.

**Proposed change:**

- When a snapshot lookup fails and `snapshot-loading-mode=refs` is in use, the catalog should detect that the metadata may be incomplete and trigger a follow-up `loadTable` call with `?snapshots=all` to obtain the full snapshot list.
- Alternatively, expose a mechanism for callers to explicitly request a full reload.
- The error returned to the caller should distinguish "snapshot absent from partial metadata" from "snapshot genuinely does not exist".

**Spec reference:**
REST Catalog Open API – loadTable:
https://redocly.github.io/redoc/?url=https://raw.githubusercontent.com/apache/iceberg/apache-iceberg-1.10.1/open-api/rest-catalog-open-api.yaml#tag/Catalog-API/operation/loadTable

**Related:**
- #1748 — added `snapshot-loading-mode` catalog property
- apache/iceberg-go#1749 — implementation PR; lazy loading deferred as follow-up

**Willingness to contribute**

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

### Proposal document

_No response_

### Specifications

- [x] Table
- [ ] View
- [x] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other

Contributor guide

Open the contributing guide

Research direction

Start at the REST catalog's LoadTable path and the snapshot-loading-mode implementation from #1748; compare its behavior with Java's RESTSessionCatalog snapshotsSupplier. Done means a refs load can obtain snapshots=all when an unreferenced snapshot is needed, while genuinely absent snapshots produce a distinct error, with REST behavior covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.