apache / apache/iceberg-rust

iceberg-catalog-glue pushes all historical schema fields to Glue, causing duplicate columns in Glue UI

Open Beginner friendly
#2,749 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

None

### Describe the bug

When using `iceberg-catalog-glue`, every call to `update_table` pushes ALL historical schema versions' fields to AWS Glue's column definition. This causes Glue Console to display duplicate columns — one entry per schema version for every field that existed in any past schema.

### To Reproduce

1. Create an Iceberg table via the Glue catalog
2. Insert data with new fields, triggering schema evolution (AddSchema + SetCurrentSchema)
3. Observe the table in AWS Glue Console → Table → Schema

### Expected behavior

Only the current schema's fields should be pushed to Glue's column definition. Historical schema versions are already preserved in the Iceberg metadata file on S3.

https://github.com/apache/iceberg-rust/blob/main/crates/catalog/glue/src/schema.rs#L57-L63
```rust
for schema in metadata.schemas_iter() {
if schema.schema_id() == current_schema.schema_id() {
continue;
}
visit_schema(schema, &mut builder)?;
}
```
The from_iceberg function visits the current schema (marked as current=true), then iterates over all historical schemas and visits them too (marked as current=false). All of these are included in the Glue TableInput.storage_descriptor.columns. Glue's API does not filter by iceberg.field.current, so the UI displays all of them.

### Willingness to contribute

None

Contributor guide

Open the contributing guide

Research direction

Start in crates/catalog/glue/src/schema.rs at from_iceberg and the visit_schema loop around lines 57-63. Verify how current and historical schemas become Glue TableInput.storage_descriptor.columns; done means Glue receives only the current schema's fields while historical schemas remain in Iceberg metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.