v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression)
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 121
Description
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
## Environment
* **iceberg-go**: v0.3.1
* **Catalog**: REST catalog (standard Iceberg REST API)
* **Go**: 1.21+
* **OS**: Linux/Windows (reproducible across)
## Summary
In v0.3.0, `table.NewAddSchemaUpdate(*Schema, lastColumnID, initial)` allowed callers to ensure the table’s `last-assigned-field-id` stayed **monotonic** even when deleting the column that previously had the highest field ID.
In **v0.3.1**, the API changed to `table.NewAddSchemaUpdate(*Schema)` (no `lastColumnID`). When we **only delete the highest-ID column(s)** and **add no new columns**, the library appears to derive `last-assigned-field-id` from the **new schema’s max field id**, which **decreases**. The REST catalog then rejects the commit with:
```
invalid_metadata: The specified metadata is not valid
```
Deleting a first/middle column works; deleting the **tail (max-ID) column** fails.
> Note: This repro excludes partition/sort references (i.e., we are not deleting a column referenced by the default spec or sort order).
## Steps to Reproduce
1. Start with a table whose current schema has fields, e.g.:
* `a` (id=1), `b` (id=2), `c` (id=3).
No partition/sort references to `c`.
2. Build a **new schema** that **removes `c`** and keeps `a`/`b` with the **same field IDs** (we don’t touch IDs).
3. Submit **two updates in one commit**:
* `AddSchema` using `table.NewAddSchemaUpdate(newSchema)`
* `SetCurrentSchema` using `table.NewSetCurrentSchemaUpdate(newSchemaID)`
(We obtain `newSchemaID` by running `b := table.MetadataBuilderFromBase(meta); id, _ := b.AddSchema(newSchema)`.)
4. Include concurrency requirements (optional but recommended):
* `AssertTableUUID(meta.UUID())`
* `AssertLastAssignedFieldID(oldLastID)` where `oldLastID` is **3** in this example.
5. `CommitTable(...)` → **fails** with `invalid_metadata`.
Minimal code sketch (v0.3.1 style):
```go
meta := tbl.Metadata()
oldLast := highestID(tbl.Schema()) // returns 3 in the example
// Build new schema that keeps a(id=1), b(id=2) only (delete c(id=3))
newSchema := buildSchemaKeepAB(tbl.Schema()) // preserves existing IDs
// Precompute new schema-id
b := table.MetadataBuilderFromBase(meta)
newSchemaID, err := b.AddSchema(newSchema)
if err != nil { panic(err) }
// Prepare updates (v0.3.1 API)
add := table.NewAddSchemaUpdate(newSchema) // no lastColumnID parameter anymore
set := table.NewSetCurrentSchemaUpdate(newSchemaID)
reqs := []table.Requirement{
table.AssertTableUUID(meta.UUID()),
table.AssertLastAssignedFieldID(int(oldLast)), // oldLast == 3
}
_, _, err = cat.CommitTable(ctx, tbl, reqs, []table.Update{add, set})
// => invalid_metadata when only deleting tail/highest-ID columns
```
## Expected Behavior
* Deleting columns (including the highest-ID column) should be allowed as long as:
* We do not change existing field IDs of the kept columns.
* `last-assigned-field-id` **does not decrease** (i.e., remains the previous value).
* In v0.3.0, passing `lastColumnID=oldLast` ensured monotonicity and commits succeeded.
## Actual Behavior
* With v0.3.1, `NewAddSchemaUpdate` cannot accept `lastColumnID`.
* When we only delete the max-ID column and add no new columns, the commit is rejected with `invalid_metadata`—apparently because the derived `last-assigned-field-id` regresses to the new schema’s max ID.
## Analysis
* Iceberg requires `last-assigned-field-id` to be **monotonic** (never decreases).
* In the “delete-tail-columns only” scenario, the **current** `last-assigned-field-id` is the old max (e.g., 3). The **new schema’s** max becomes smaller (e.g., 2).
If the client or server infers the counter from the new schema’s max, it violates monotonicity → `invalid_metadata`.
## Workarounds
* **Add a sentinel (dummy) column** in the same update with ID = `oldLast + 1` (e.g., `__compat_padding_...`), nullable, never used. This keeps the new schema’s max ≥ old max.
Or, more practically, add a real new column in the same change so max ID increases.
* (Less ideal) Maintain a fork that restores the older API (`NewAddSchemaUpdate(schema, lastColumnID, initial)`) or custom-craft the REST payload to set `last-column-id = oldLast`.
* Of course still ensure you’re not deleting a field referenced by partition spec or sort order (not the case in this repro).
## Proposal
* **API / behavior options:**
1. Re-introduce a way to set **`lastColumnID`** (or an equivalent parameter) on `AddSchema` in the Go client; or
2. Have the client compute `last-assigned-field-id` as `max(oldLastID, max(newSchema.FieldIDs))` so it never regresses; or
3. Provide a dedicated update or requirement to explicitly set/preserve `last-assigned-field-id` without requiring a dummy column.
* **Docs**: Clarify in v0.3.1 migration notes that callers must ensure the counter doesn’t regress when deleting the highest-ID column, and suggest recommended patterns.
## Additional Context
* The same flow succeeds if we delete a middle/first column (the new schema’s max ID stays the same).
* The same flow succeeds if we add at least one new column (the new schema’s max ID increases).
Happy to provide a tiny repro program if needed. Thanks!
Contributor guide
Research direction
Start by tracing table.NewAddSchemaUpdate, MetadataBuilderFromBase, and CommitTable through the schema-update and REST commit flow. Reproduce deletion of the highest-ID column and verify that the resulting metadata preserves the prior last-assigned-field-id while still allowing the schema update to commit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100