delta-io / delta-io/delta-kernel-rs

Incorporate `read_actions()` logic into the `LogReplayProcessor` trait

Open
#809 0 comments 0 reactions 0 assignees View on GitHub
feature-request
Dominant language
Rust
Stars
361
Forks
214
Avg merge
2d 17h
Merged PRs (30d)
87

Description

### Please describe why this is necessary.

The work to refactor the `CDF` `LogReplayScanner` to leverage the new `LogReplayProcessor` trait has uncovered issues https://github.com/delta-io/delta-kernel-rs/pull/796. Unlike the `CheckpointLogReplayProcessor` and the `ScanLogReplayProcessor`, CDF requires **two passes** of the iterator of actions produced from reading **a commit file**, before moving onto processing the next commit file.

With the current way the `LogReplayProcessor` trait is structured, processors should only have to ever apply one overarching processing function to a **single** iterator of actions, which can not be directly applied to the cdf case. (the best workaround to this is to have two 'processors' that apply processing for each iteration of actions from a single commit file)

To get around this, we can embed `read_actions()` logic into the `LogReplayProcessor` trait, so the replay processor now performs log replay over a `LogSegment` instead. This way we have 2 functions exposed to processors, one that goes over a single batch, and one that goes over the entire log segment.

### Describe the functionality you are proposing.

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading the LogReplayProcessor trait, read_actions(), LogReplayScanner, and the CheckpointLogReplayProcessor and ScanLogReplayProcessor implementations, with PR 796 for context. Trace how CDF needs two passes over actions from one commit file. Done means replay operates on a LogSegment while processors can handle both one batch and the full segment.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.