updateTable using glue with a large schema fails
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 121
Description
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
When using glue and a large message schema we get this error on every updateCatalog
Glue: UpdateTable, https response error StatusCode: 400, RequestID: 21d099dd-a972-488f-b3ab-7a6a2a1a4c35, InvalidInputException: Payload size of request exceeded limit
I believe it's because constructTableInput always includes all schema columns via schemasToGlueColumns(staged.Metadata()) on every UpdateTable call. We have roughly 3600 fields. This is by design because we use these records to write to elastic search with the "elastic common schema" format which has a lot of fields. A usual message will only have between 20-50 fields set but all fields exist in the schema.
Would it make sense to be able to avoid having the schema in updateTable? I believe it's only used as metadata used in athena, the real schema would still exist in the metadata and parquet files?
Contributor guide
Research direction
Start at constructTableInput and the schemasToGlueColumns(staged.Metadata()) call used by updateCatalog. Inspect how Glue UpdateTable requests are assembled and how the schema is used for Athena versus Iceberg metadata. Done means updates for large schemas no longer exceed Glue's payload limit while preserving the real schema in the metadata and Parquet files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100