TimelyDataflow / TimelyDataflow/differential-dataflow

`wrapper/frontier.rs` does not update metadata

Open
#271 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

The TraceFrontier trace wrapper is used to forcibly advance timestamps in a trace, to bring data up to some common frontier (often because compaction may have occurred, and we want to ensure rather than hope that data have been advanced). However, the wrapper does not update the metadata about valid times, and in particular just mirrors the trace's advance_frontier and each batch's since frontier. Both of these should probably be lower-bounded by the frontier used to construct the wrapper.

I believe most operators are agnostic to these frontiers, and so this is probably not the source of any bugs of that flavor. However, it seems possible that operators should be less agnostic to them in the future (or now) and having incorrect metadata is never good.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in wrapper/frontier.rs with TraceFrontier, then trace how advance_frontier and each batch's since frontier are mirrored. Compare those metadata values with the frontier used to construct the wrapper, and update the behavior so the metadata reflects that lower bound. Verify the wrapper's frontier and batch metadata remain consistent after advancement.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.