iceberg-catalog-glue pushes all historical schema fields to Glue, causing duplicate columns in Glue UI
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
None
### Describe the bug
When using `iceberg-catalog-glue`, every call to `update_table` pushes ALL historical schema versions' fields to AWS Glue's column definition. This causes Glue Console to display duplicate columns — one entry per schema version for every field that existed in any past schema.
### To Reproduce
1. Create an Iceberg table via the Glue catalog
2. Insert data with new fields, triggering schema evolution (AddSchema + SetCurrentSchema)
3. Observe the table in AWS Glue Console → Table → Schema
### Expected behavior
Only the current schema's fields should be pushed to Glue's column definition. Historical schema versions are already preserved in the Iceberg metadata file on S3.
https://github.com/apache/iceberg-rust/blob/main/crates/catalog/glue/src/schema.rs#L57-L63
```rust
for schema in metadata.schemas_iter() {
if schema.schema_id() == current_schema.schema_id() {
continue;
}
visit_schema(schema, &mut builder)?;
}
```
The from_iceberg function visits the current schema (marked as current=true), then iterates over all historical schemas and visits them too (marked as current=false). All of these are included in the Glue TableInput.storage_descriptor.columns. Glue's API does not filter by iceberg.field.current, so the UI displays all of them.
### Willingness to contribute
None
Contributor guide
Research direction
Start in crates/catalog/glue/src/schema.rs at from_iceberg and the visit_schema loop around lines 57-63. Verify how current and historical schemas become Glue TableInput.storage_descriptor.columns; done means Glue receives only the current schema's fields while historical schemas remain in Iceberg metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100