Support delete-file-backed row changes in incremental changelog scans
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 116
Description
## Background
Incremental changelog scanning was added in [#1883](https://github.com/apache/iceberg-go/pull/1883) to mirror Java's `newIncrementalChangelogScan()` API.
The first implementation handles file-level changes from data manifests. This issue tracks delete-file-backed row changes and keeps the related work together.
Current status: file-level changelog planning is implemented in #1883. Snapshots with delete manifests are still rejected.
## Implemented
- [x] `Table.NewIncrementalChangelogScan`
- [x] `INSERT` tasks for added data files
- [x] `DELETE` tasks for removed data files
- [x] Snapshot boundaries, change ordinals, and commit snapshot IDs
- [x] Filters, residuals, row-lineage task fields, and local planning
## Planned
- [ ] Define changelog task and reader types for added rows, deleted data files, and deleted rows
- [ ] Plan position and equality delete files across snapshot ranges
- [ ] Preserve added versus existing delete-file semantics
- [ ] Support deletion vectors and v3 row lineage
- [ ] Add end-to-end tests for same-snapshot changes, prior deletes, file removal, schema evolution, filters, and boundaries
- [ ] Add documentation and decide remote planning support
## Implementation notes
- The normal scan path removes deleted rows while reading. A changelog scan needs to emit rows removed by the current snapshot.
- A deleted data file must be read with delete files that existed before the file was removed.
- A row-level delete needs to distinguish delete files added by the current snapshot from delete files that were already present.
- The task shape should follow Java's `AddedRowsScanTask`, `DeletedDataFileScanTask`, and `DeletedRowsScanTask`, with Go naming.
- Position-delete, equality-delete, and deletion-vector readers already exist in the table scan and inspection paths. This work should reuse them where possible.
- Remote planning can remain unsupported initially.
## Related work
- [PR #1883](https://github.com/apache/iceberg-go/pull/1883)
- [Table.Inspect metadata table support #1703](https://github.com/apache/iceberg-go/issues/1703) (related groundwork only)
- [Delete-file metadata support #1699](https://github.com/apache/iceberg-go/pull/1699)
- [Position deletes metadata support #1747](https://github.com/apache/iceberg-go/pull/1747)
## References
- [Java changelog planner](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java)
- [Java changelog task types](https://github.com/apache/iceberg/tree/main/api/src/main/java/org/apache/iceberg)
- [Iceberg table specification](https://iceberg.apache.org/spec/)
Contributor guide
Research direction
Start with Table.NewIncrementalChangelogScan and the existing delete-file readers in the table scan and inspection paths. Compare the Java changelog planner and task types, then define Go task and reader types, plan position and equality deletes across snapshot ranges, and add the listed end-to-end coverage for snapshot changes, filters, schema evolution, and boundaries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100