apache / apache/iceberg

Error while executing "rewrite_position_delete_files"

Open
#14,619 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.10.0 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

We have an error when calling "rewrite_position_delete_files"

The call happens in a Spark (3.5.5) session:
`spark.sql("CALL iceberg.system.rewrite_position_delete_files( table => 'lake.table', options => map('rewrite-all', 'true'))");`

Starting with Iceberg 1.10.0 we have the following error:
`Caused by: org.apache.iceberg.exceptions.ValidationException: Invalid partition field parent: list
at org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:49)
at org.apache.iceberg.PartitionSpec.checkCompatibility(PartitionSpec.java:672)
at org.apache.iceberg.PartitionSpec$Builder.build(PartitionSpec.java:628)
at org.apache.iceberg.PartitionSpec$Builder.build(PartitionSpec.java:623)
at org.apache.iceberg.expressions.ExpressionUtil.identitySpec(ExpressionUtil.java:656)
at org.apache.iceberg.expressions.ExpressionUtil.extractByIdInclusive(ExpressionUtil.java:156)
at org.apache.iceberg.spark.source.PositionDeletesRowReader.open(PositionDeletesRowReader.java:95)
at org.apache.iceberg.spark.source.PositionDeletesRowReader.open(PositionDeletesRowReader.java:44)`

Our tables had a partition layout change long ago and the partition spec currently looks like:
`
"default-spec-id" : 1,
"partition-specs" : [ {
"spec-id" : 0,
"fields" : [ {
"name" : "is_ok",
"transform" : "identity",
"source-id" : 38,
"field-id" : 1000
}, {
"name" : "timestamp_hour",
"transform" : "hour",
"source-id" : 2,
"field-id" : 1001
} ]
}, {
"spec-id" : 1,
"fields" : [ {
"name" : "timestamp_hour",
"transform" : "hour",
"source-id" : 2,
"field-id" : 1001
} ]
} ]
`
The code seems to break on the old "is_ok" partition.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with the supplied Spark CALL and the stack-trace entry point PositionDeletesRowReader.open, then follow ExpressionUtil.extractByIdInclusive and identitySpec into PartitionSpec.Builder.build and checkCompatibility. Reproduce the failure with the shown evolved partition specs; done means rewrite_position_delete_files handles that metadata without the ValidationException and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.