apache / apache/iceberg

REST: Add query parameters to trim snapshot-log and metadata-log in loadTable responses

Open
#15,947 0 comments 0 reactions 0 assignees View on GitHub
proposal
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Proposed Change

The `loadTable` response includes the full `TableMetadata` object, which contains two unbounded arrays that grow with every commit:

- **`snapshot-log`** (`SnapshotLog` schema, [rest-catalog-open-api.yaml, line ~2653](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L2653)): An array of `{snapshot-id, timestamp-ms}` entries — one per snapshot operation.
- **`metadata-log`** (`MetadataLog` schema, line ~2668): An array of `{metadata-file, timestamp-ms}` entries — one per metadata file change.

For long-lived, high-write tables these arrays can contain thousands of entries, significantly inflating the `loadTable` response. Unlike the snapshot objects themselves (which can be filtered via the existing `?snapshots=refs|all` query parameter, line ~975), there is **no mechanism** to limit or omit these log arrays.

In practice, most consumers of `loadTable` need only the current table state — current snapshot, schema, partition spec, and sort order. The snapshot-log and metadata-log are primarily useful for time-travel queries and metadata auditing, not routine table operations. This is closely related to the broader concern raised in [#14877](https://github.com/apache/iceberg/issues/14877) about `TableMetadata` size, where a committer noted that the REST spec direction is to let catalogs control what they return rather than requiring the full blob.

**Proposed spec changes:**

Add two optional query parameters to the `loadTable` GET operation:

```yaml
- in: query
name: snapshot-log
description:
Controls whether the snapshot-log array is included in the response metadata.
Setting the value to `none` omits the snapshot-log entirely.
Setting the value to `all` returns the full snapshot-log (default behavior).
required: false
schema:
type: string
enum: [all, none]

- in: query
name: metadata-log
description:
Controls whether the metadata-log (previous metadata file entries) is included
in the response metadata. Setting the value to `none` omits the metadata-log
entirely. Setting the value to `all` returns the full metadata-log (default behavior).
required: false
schema:
type: string
enum: [all, none]
```

The defaults are `all` to preserve backward compatibility. Setting `none` causes the server to return an empty array `[]` for that field.

**Alternative considered:** `latest-N` enum values (e.g., `snapshot-log=latest-10`) were considered but rejected in this proposal to keep the parameter simple and aligned with the existing `snapshots` parameter pattern (`all` vs a reduced mode). A numeric limit could be a follow-up if there is demand.

**Backward compatibility:**

- When neither parameter is provided, behavior is identical to today (full logs returned).
- Servers that don't recognize the parameters return the full response (safe fallback).
- Clients that don't send the parameters see no change.

**Willingness to contribute**

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

### Proposal document

_No response_

### Specifications

- [x] Table
- [ ] View
- [x] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other

Contributor guide

Open the contributing guide

Research direction

Start with the loadTable GET operation around line 975 in rest-catalog-open-api.yaml and the SnapshotLog and MetadataLog schemas around lines 2653 and 2668. Add the two optional enum parameters with their documented defaults and verify that the specification describes omission as returning empty arrays while preserving existing behavior when parameters are absent.

Written by the indexing model from the issue text.

Assessment

Tech stack
openapi
Domain
api, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.