apache / apache/iceberg

Spark RewriteManifestsSparkAction sortBy throws "AnalysisException: [FIELD_NOT_FOUND] No such struct field" for nested field in partition spec

Open Beginner friendly
#17,425 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

In Spark 4.1, calling rewrite manifests procedure with sort by on a partition field that have a dot in the name after being flattened (for example a nested field in a struct), will throw this exception
`AnalysisException: [FIELD_NOT_FOUND] No such struct field a in a.b`
This is because in [RewriteManifestsSpark::sortColumn](https://github.com/apache/iceberg/blob/f3d5b1768d7e3939cf5d43b0f943cfdca24ceca2/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java#L320) builds the Spark column reference for a custom clustering field by string concatenation — col("data_file.partition." + partitionFieldName), we could fix it by quotting `partitionFieldName`

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start in spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java at sortColumn, especially around line 320, and reproduce the Spark 4.1 rewrite manifests procedure with a nested partition field whose flattened name contains a dot. Done means sorting by that partition field no longer raises FIELD_NOT_FOUND.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.