TimelyDataflow / TimelyDataflow/differential-dataflow

Arrange operator does not compact trace on dataflow shutdown.

Open
#250 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

The arrange operator compacts traces as part of its scheduling work. However, if the dataflow graph is shut down, for example because all inputs close and work completes, the arrange operator will no longer be scheduled and so will no longer compact the representation.

This is only a problem in the case that DIFFERENTIAL_EAGER_MERGE is set, and a trace is captured from a dataflow. In that case, one might expect the background compaction to continue to flatten the captured trace. Instead, it is frozen in whatever form it had at dataflow close time.

It would be reasonable for the arrange operator to report that it has not completed (in the timely dataflow sense of the return value from schedule) as long as outstanding merge work exists. The downside is that this could keep the dataflow live longer than is needed, but in this case it should be easy enough to drop any held references to the trace (the arrange operator should only require merge effort if the trace is still held, as it holds only a weak reference).

cc @ruchirk

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the arrange operator's scheduling implementation, focusing on schedule's return value, DIFFERENTIAL_EAGER_MERGE, and weak trace reference handling. Inspect shutdown with all inputs closed and a captured trace; done when outstanding merge work keeps the dataflow live until compaction completes, while dropping the trace allows shutdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.