apache / apache/iceberg-go

discussion: align v1/v2 metadata parsing with Java/PyIceberg on v3-only fields

Open
#1,086 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
463
Forks
232
Avg merge
2d 11h
Merged PRs (30d)
121

Description

Follow-up from #1069, open question.

#1069 hard-errors on `next-row-id` / `encryption-keys` in v1/v2 metadata. None of the reference clients do this — Java reads `next-row-id` only when `formatVersion >= 3` and defaults otherwise, has no version guard on `encryption-keys` at all; PyIceberg and iceberg-rust both drop unknown fields. So we're the only one that rejects here.

Doesn't bite today because v3 isn't widely deployed. Once downgrade tooling shows up, a v2 file with stray v3 fields reads everywhere except Go.

Strict surface has value (mis-versioned metadata is by definition wrong, and it catches our own writer if it ever leaks v3 fields), but forward-compat is also a spec goal. If we keep it strict, a code comment near `rejectV3OnlyFields` capturing the intentional divergence would help future readers not read it as a parity bug.

Contributor guide

Open the contributing guide

Research direction

Start with #1069 and the metadata parsing path around `rejectV3OnlyFields`; compare the current behavior with the Java, PyIceberg, and iceberg-rust behavior described here. Done means resolving the strictness versus forward-compatibility question and recording the chosen behavior, including a comment near the guard if strict rejection remains.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.