Error while executing "rewrite_position_delete_files"
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
We have an error when calling "rewrite_position_delete_files"
The call happens in a Spark (3.5.5) session:
`spark.sql("CALL iceberg.system.rewrite_position_delete_files( table => 'lake.table', options => map('rewrite-all', 'true'))");`
Starting with Iceberg 1.10.0 we have the following error:
`Caused by: org.apache.iceberg.exceptions.ValidationException: Invalid partition field parent: list
at org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:49)
at org.apache.iceberg.PartitionSpec.checkCompatibility(PartitionSpec.java:672)
at org.apache.iceberg.PartitionSpec$Builder.build(PartitionSpec.java:628)
at org.apache.iceberg.PartitionSpec$Builder.build(PartitionSpec.java:623)
at org.apache.iceberg.expressions.ExpressionUtil.identitySpec(ExpressionUtil.java:656)
at org.apache.iceberg.expressions.ExpressionUtil.extractByIdInclusive(ExpressionUtil.java:156)
at org.apache.iceberg.spark.source.PositionDeletesRowReader.open(PositionDeletesRowReader.java:95)
at org.apache.iceberg.spark.source.PositionDeletesRowReader.open(PositionDeletesRowReader.java:44)`
Our tables had a partition layout change long ago and the partition spec currently looks like:
`
"default-spec-id" : 1,
"partition-specs" : [ {
"spec-id" : 0,
"fields" : [ {
"name" : "is_ok",
"transform" : "identity",
"source-id" : 38,
"field-id" : 1000
}, {
"name" : "timestamp_hour",
"transform" : "hour",
"source-id" : 2,
"field-id" : 1001
} ]
}, {
"spec-id" : 1,
"fields" : [ {
"name" : "timestamp_hour",
"transform" : "hour",
"source-id" : 2,
"field-id" : 1001
} ]
} ]
`
The code seems to break on the old "is_ok" partition.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with the supplied Spark CALL and the stack-trace entry point PositionDeletesRowReader.open, then follow ExpressionUtil.extractByIdInclusive and identitySpec into PartitionSpec.Builder.build and checkCompatibility. Reproduce the failure with the shown evolved partition specs; done means rewrite_position_delete_files handles that metadata without the ValidationException and the behavior is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100