Spark RewriteManifestsSparkAction sortBy throws "AnalysisException: [FIELD_NOT_FOUND] No such struct field" for nested field in partition spec
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
In Spark 4.1, calling rewrite manifests procedure with sort by on a partition field that have a dot in the name after being flattened (for example a nested field in a struct), will throw this exception
`AnalysisException: [FIELD_NOT_FOUND] No such struct field a in a.b`
This is because in [RewriteManifestsSpark::sortColumn](https://github.com/apache/iceberg/blob/f3d5b1768d7e3939cf5d43b0f943cfdca24ceca2/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java#L320) builds the Spark column reference for a custom clustering field by string concatenation — col("data_file.partition." + partitionFieldName), we could fix it by quotting `partitionFieldName`
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start in spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java at sortColumn, especially around line 320, and reproduce the Spark 4.1 rewrite manifests procedure with a nested partition field whose flattened name contains a dot. Done means sorting by that partition field no longer raises FIELD_NOT_FOUND.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100