TimelyDataflow / TimelyDataflow/differential-dataflow
Audit operator code for "overload protection"
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
At least one operator (https://github.com/TimelyDataflow/differential-dataflow/pull/301) has an implementation that risks operators getting wedged if they process inputs more slowly than they are produced upstream. Ideally, operators would ensure they perform a finite amount of work each time they are scheduled, but I suspect that many risk continuing to run as long as input data are available.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the operator implementation discussed in pull request #301, then identify the other operator entry points that process upstream inputs. Audit whether each scheduled invocation performs finite work, and document or address any operators that can continue running while input remains available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100