apache / apache/iceberg-go

v0.3.1: `NewAddSchemaUpdate` no longer takes `lastColumnID` → deleting highest-ID column triggers `invalid_metadata` (last-assigned-field-id regression)

Open
#538 21 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
463
Forks
232
Avg merge
2d 11h
Merged PRs (30d)
121

Description

### Apache Iceberg version

main (development)

### Please describe the bug 🐞

## Environment

* **iceberg-go**: v0.3.1
* **Catalog**: REST catalog (standard Iceberg REST API)
* **Go**: 1.21+
* **OS**: Linux/Windows (reproducible across)

## Summary

In v0.3.0, `table.NewAddSchemaUpdate(*Schema, lastColumnID, initial)` allowed callers to ensure the table’s `last-assigned-field-id` stayed **monotonic** even when deleting the column that previously had the highest field ID.

In **v0.3.1**, the API changed to `table.NewAddSchemaUpdate(*Schema)` (no `lastColumnID`). When we **only delete the highest-ID column(s)** and **add no new columns**, the library appears to derive `last-assigned-field-id` from the **new schema’s max field id**, which **decreases**. The REST catalog then rejects the commit with:

```
invalid_metadata: The specified metadata is not valid
```

Deleting a first/middle column works; deleting the **tail (max-ID) column** fails.

> Note: This repro excludes partition/sort references (i.e., we are not deleting a column referenced by the default spec or sort order).

## Steps to Reproduce

1. Start with a table whose current schema has fields, e.g.:

* `a` (id=1), `b` (id=2), `c` (id=3).
No partition/sort references to `c`.

2. Build a **new schema** that **removes `c`** and keeps `a`/`b` with the **same field IDs** (we don’t touch IDs).

3. Submit **two updates in one commit**:

* `AddSchema` using `table.NewAddSchemaUpdate(newSchema)`
* `SetCurrentSchema` using `table.NewSetCurrentSchemaUpdate(newSchemaID)`
(We obtain `newSchemaID` by running `b := table.MetadataBuilderFromBase(meta); id, _ := b.AddSchema(newSchema)`.)

4. Include concurrency requirements (optional but recommended):

* `AssertTableUUID(meta.UUID())`
* `AssertLastAssignedFieldID(oldLastID)` where `oldLastID` is **3** in this example.

5. `CommitTable(...)` → **fails** with `invalid_metadata`.

Minimal code sketch (v0.3.1 style):

```go
meta := tbl.Metadata()
oldLast := highestID(tbl.Schema()) // returns 3 in the example

// Build new schema that keeps a(id=1), b(id=2) only (delete c(id=3))
newSchema := buildSchemaKeepAB(tbl.Schema()) // preserves existing IDs

// Precompute new schema-id
b := table.MetadataBuilderFromBase(meta)
newSchemaID, err := b.AddSchema(newSchema)
if err != nil { panic(err) }

// Prepare updates (v0.3.1 API)
add := table.NewAddSchemaUpdate(newSchema) // no lastColumnID parameter anymore
set := table.NewSetCurrentSchemaUpdate(newSchemaID)

reqs := []table.Requirement{
table.AssertTableUUID(meta.UUID()),
table.AssertLastAssignedFieldID(int(oldLast)), // oldLast == 3
}
_, _, err = cat.CommitTable(ctx, tbl, reqs, []table.Update{add, set})
// => invalid_metadata when only deleting tail/highest-ID columns
```

## Expected Behavior

* Deleting columns (including the highest-ID column) should be allowed as long as:

* We do not change existing field IDs of the kept columns.
* `last-assigned-field-id` **does not decrease** (i.e., remains the previous value).
* In v0.3.0, passing `lastColumnID=oldLast` ensured monotonicity and commits succeeded.

## Actual Behavior

* With v0.3.1, `NewAddSchemaUpdate` cannot accept `lastColumnID`.
* When we only delete the max-ID column and add no new columns, the commit is rejected with `invalid_metadata`—apparently because the derived `last-assigned-field-id` regresses to the new schema’s max ID.

## Analysis

* Iceberg requires `last-assigned-field-id` to be **monotonic** (never decreases).
* In the “delete-tail-columns only” scenario, the **current** `last-assigned-field-id` is the old max (e.g., 3). The **new schema’s** max becomes smaller (e.g., 2).
If the client or server infers the counter from the new schema’s max, it violates monotonicity → `invalid_metadata`.

## Workarounds

* **Add a sentinel (dummy) column** in the same update with ID = `oldLast + 1` (e.g., `__compat_padding_...`), nullable, never used. This keeps the new schema’s max ≥ old max.
Or, more practically, add a real new column in the same change so max ID increases.
* (Less ideal) Maintain a fork that restores the older API (`NewAddSchemaUpdate(schema, lastColumnID, initial)`) or custom-craft the REST payload to set `last-column-id = oldLast`.
* Of course still ensure you’re not deleting a field referenced by partition spec or sort order (not the case in this repro).

## Proposal

* **API / behavior options:**

1. Re-introduce a way to set **`lastColumnID`** (or an equivalent parameter) on `AddSchema` in the Go client; or
2. Have the client compute `last-assigned-field-id` as `max(oldLastID, max(newSchema.FieldIDs))` so it never regresses; or
3. Provide a dedicated update or requirement to explicitly set/preserve `last-assigned-field-id` without requiring a dummy column.

* **Docs**: Clarify in v0.3.1 migration notes that callers must ensure the counter doesn’t regress when deleting the highest-ID column, and suggest recommended patterns.

## Additional Context

* The same flow succeeds if we delete a middle/first column (the new schema’s max ID stays the same).
* The same flow succeeds if we add at least one new column (the new schema’s max ID increases).

Happy to provide a tiny repro program if needed. Thanks!

Contributor guide

Open the contributing guide

Research direction

Start by tracing table.NewAddSchemaUpdate, MetadataBuilderFromBase, and CommitTable through the schema-update and REST commit flow. Reproduce deletion of the highest-ID column and verify that the resulting metadata preserves the prior last-assigned-field-id while still allowing the schema update to commit.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.