perf(table): avoid full snapshot materialization when creating metadata builders
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 121
Description
### Proposed Change
#1956 adds deferred materialization of unreferenced snapshot history when a REST commit response is parsed with `snapshot-loading-mode=refs`. The benefit is lost when the returned table is used to start another transaction.
`Table.NewTransaction*` calls `MetadataBuilderFromBase`, which currently requests the complete snapshot collection. This immediately materializes every deferred historical snapshot, even though the normal REST commit path primarily needs the accumulated requirements and updates rather than a fully rebuilt metadata document.
This path pays both the deferred indexing cost and the later full-decoding cost. It is therefore more expensive than eager parsing when another transaction is created immediately.
Measured on an Apple M1 Max using medians of five runs:
| Metadata | Eager parse + builder | Deferred parse + builder | Time | Allocations | Allocated bytes |
|---:|---:|---:|---:|---:|---:|
| 1,999,830 B | 80.611 ms | 98.314 ms | +22.0% | +20.7% | +38.9% |
| 5,856,882 B | 232.241 ms | 287.704 ms | +23.9% | +20.7% | +38.5% |
This is a performance limitation, not a known correctness problem. It is intentionally separate from #1946, whose scope is commit-response parsing.
**Potential approaches:**
This issue does not select a design. The following alternatives should be evaluated independently:
- **Short term:** Delay complete snapshot materialization during `MetadataBuilderFromBase` construction. Materialize only when a builder operation actually requires the complete snapshot collection.
- **Mid term:** Represent builder snapshot changes as mutations over an immutable deferred base, allowing additions, removals, ref validation, and individual ID lookup without decoding unrelated historical snapshots.
- **Long term:** Introduce a shared snapshot-collection abstraction used by metadata and metadata builders, with consistent support for indexed lookup, lazy iteration or materialization, mutation, cloning, validation, and serialization.
**Correctness requirements:**
- Preserve eager and deferred metadata behavior and parse-time validation semantics.
- Preserve complete snapshot history across metadata building, serialization, equality, and v1/v2/v3 conversion.
- Builder additions and removals must not lose, duplicate, or resurrect deferred snapshots.
- Ref and current-snapshot validation must account for both materialized and deferred snapshots.
- Preserve defensive-copy and concurrency guarantees.
- Default behavior remains unchanged for metadata without deferred snapshot state.
**Performance validation:**
Add benchmarks at multiple metadata sizes that measure, at minimum:
- eager parse followed by `MetadataBuilderFromBase`;
- deferred parse followed by `MetadataBuilderFromBase`;
- representative snapshot addition and ref-update transactions;
- snapshot removal or expiration operations;
- full metadata build and serialization.
The intended result is to avoid full history decoding for REST transactions that do not require it, without making operations that genuinely require the complete collection materially worse.
**Backward compatibility:**
- No breaking public API changes should be required.
- Full snapshot history must remain available and must never be silently truncated.
- Existing metadata validation and error behavior must be preserved.
- Eagerly parsed metadata should continue to use the existing builder behavior unless a common abstraction is adopted.
**Related:**
- #1946 — defer unreferenced snapshot materialization while parsing REST commit responses
- #1956 — implementation of the deferred commit-response parsing path
**Willingness to contribute**
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
### Proposal document
_Not applicable — client-side performance optimization._
### Specifications
- [x] Table
- [ ] View
- [x] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other
Contributor guide
Research direction
Start at Table.NewTransaction* and MetadataBuilderFromBase, then trace how deferred snapshot state is requested from a REST commit response. Compare the listed eager and deferred builder benchmarks, and define completion as preserving validation, history, serialization, conversions, and builder mutations while meeting the stated performance goals.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100