delta-io / delta-io/delta-kernel-rs
Incorporate `read_actions()` logic into the `LogReplayProcessor` trait
- Dominant language
- Rust
- Stars
- 361
- Forks
- 214
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 87
Description
### Please describe why this is necessary.
The work to refactor the `CDF` `LogReplayScanner` to leverage the new `LogReplayProcessor` trait has uncovered issues https://github.com/delta-io/delta-kernel-rs/pull/796. Unlike the `CheckpointLogReplayProcessor` and the `ScanLogReplayProcessor`, CDF requires **two passes** of the iterator of actions produced from reading **a commit file**, before moving onto processing the next commit file.
With the current way the `LogReplayProcessor` trait is structured, processors should only have to ever apply one overarching processing function to a **single** iterator of actions, which can not be directly applied to the cdf case. (the best workaround to this is to have two 'processors' that apply processing for each iteration of actions from a single commit file)
To get around this, we can embed `read_actions()` logic into the `LogReplayProcessor` trait, so the replay processor now performs log replay over a `LogSegment` instead. This way we have 2 functions exposed to processors, one that goes over a single batch, and one that goes over the entire log segment.
### Describe the functionality you are proposing.
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reading the LogReplayProcessor trait, read_actions(), LogReplayScanner, and the CheckpointLogReplayProcessor and ScanLogReplayProcessor implementations, with PR 796 for context. Trace how CDF needs two passes over actions from one commit file. Done means replay operates on a LogSegment while processors can handle both one batch and the full segment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100