TimelyDataflow / TimelyDataflow/timely-dataflow
Investigate unifying `input` with `replay`, and `probe` with `capture`.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 293
- Avg merge
- 14h 46m
- Merged PRs (30d)
- 4
Description
We have two general methods for introducing and exfiltrating timely streams, replay and capture respectively. At the same time, we have custom operators for input and probe, which introduce input from and report progress back to external sources. It seems like these could be unified, though perhaps there are unforeseen complications.
Similarly, to_stream seems like it could be unified with either replay or input, or ideally both. I think at the moment there is a blanket impl issue, in that the source for replay implements Iterator with an Event item type, rather than whatever data type we actually want the stream to present.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin by comparing the implementations and interfaces of replay, capture, input, probe, and to_stream, especially the Iterator item type used by replay. Trace the blanket implementation concern and determine whether these operators can share a coherent abstraction; done means documenting a settled unification design or the complications that prevent it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100