DynamicWriteResultSerializer version not bumped after format change
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Flink
### Please describe the bug 🐞
`DynamicWriteResultSerializer` was modified in #14810 to add a `specId` field to the serialized format, but `getVersion()` was not bumped from `1`. This breaks the `SimpleVersionedSerializer` versioning contract: if a checkpoint from a job running the pre-#14810 code (V1, no `specId`) is restored on the current code (still claiming V1 but expecting `specId`), the `readInt()` for `specId` will consume the first four bytes of the `WriteResult` payload, producing a corrupted `WriteResult` and silent data loss.
### Steps to Reproduce
1. Start a Flink job using the dynamic Iceberg sink on a version before #14810 (serializes `WriteTarget` + `WriteResult` bytes as V1).
2. Take a checkpoint.
3. Upgrade the job to a version with #14810 (serializes `TableKey` + `specId` + `WriteResult` bytes, but still claims V1).
4. Restore from the checkpoint.
5. `deserialize(1, ...)` runs the new V1 code path, which calls `TableKey.deserializeFrom(view)` (reads only `tableName` + `branch`) then `view.readInt()` for `specId` — but the old V1 bytes have `schemaId` (an int) immediately after `branch`, so `readInt()` consumes `schemaId` as `specId`, and the remaining bytes (starting from the real `specId` field) are passed to `WriteResultSerializer.deserialize()`, producing a corrupt `WriteResult`.
### Expected Behavior
The serializer should use distinct version numbers for the two formats so that old V1 bytes are deserialized using the old V1 layout (`WriteTarget` + `WriteResult`) and new V2 bytes are deserialized using the new V2 layout (`TableKey` + `specId` + `WriteResult`).
### Actual Behavior
Both formats claim version 1, so the deserializer cannot distinguish them, leading to silent corruption on upgrade.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start at DynamicWriteResultSerializer, especially getVersion() and deserialize(version, ...), and trace the old WriteTarget + WriteResult and new TableKey + specId + WriteResult layouts described in the issue. Verify checkpoint compatibility across both formats, with distinct versions and no corruption when restoring old bytes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100