TimelyDataflow / TimelyDataflow/differential-dataflow

Consolidate Timestamps and Time Windowed Dataflows

Open
#351 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

I've not quite got around to trying this stuff out but from reading the docs I suspect this may covered by the empty chapter found here:

https://timelydataflow.github.io/differential-dataflow/chapter_6/chapter_6.html

I have two main questions/problems I'm aiming solve.

Firstly, I'm unsure how to go about consolidating differences from higher resolution timestamps to a single, lower resolution difference. E.g. perhaps 5 separate differences occurred within a minute but I only care about knowing what happened within the minute, not exactly when within the minute. I think the same logic could apply for any kind of timestamp that was a numeric type.

Secondly, I'd like to have a dataflow retain changes and reverse them upon a specified time window passing. Essentially forming a temporal window.

In both cases, I think it would be solvable to have logic be applied to the data prior to inputting it into the dataflow. However, I'd quite like a reusable operator for each. I don't think I can achieve either outcome with existing operators but I could well be wrong. Does anyone have any ideas how this could be approached? Are operators for these outcomes going to result in some unforeseen issues?

Thanks for your time regardless of whether you can help!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the empty Chapter 6 page linked in the issue and review the existing operator documentation and examples around it. Clarify whether reusable operators for timestamp consolidation and time-window reversal are feasible, then define their behavior and acceptance criteria before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.