TimelyDataflow / TimelyDataflow/timely-dataflow
closure ergonomics with custom operators
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 293
- Avg merge
- 14h 46m
- Merged PRs (30d)
- 4
Description
I'm working on a timely program that will involve ~5 custom operators, each of which is fairly involved. Ideally I'd like to import those custom operators into main.rs for readability's sake. Something like:
input.to_stream(scope)
.map(..)
.custom_operator1()
.filter(..)
.custom_operator2()
{etc}
where the custom operators are unary_frontiers with complicated closures.
I tried to write my own traits by copying the type signatures in operators like count and filter, but I think the issue is my custom operators are not as generic as those operators and I'm getting tripped up over the type signatures. My operators assume that each record is of the form ((id1, id2, id3, id4), Vec) so I end up getting errors like
--> src/lib.rs:182:13
|
169 | fn location_aggregation(&self) -> Stream<S, D> {
| ------------ expected `operations::timely::dataflow::Stream<S, D>` because of return type
...
182 | / self.unary_frontier(
183 | | exchange,
184 | | "agg locs",
185 | | |_capability| {
... |
309 | | }
310 | | })
| |__________________________^ expected type parameter, found tuple
|
= note: expected type `operations::timely::dataflow::Stream<_, D>`
found type `operations::timely::dataflow::Stream<_, ((isize, _, _, _), std::vec::Vec<f64>)>`
I feel like this problem has more to do with my lack of rust experience than timely, so I'm not sure this is ticket-worthy. Feel free to close, if so. But do you have an example of a custom operator definition that isn't as generic as count et al?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the custom operator code in src/lib.rs and compare its unary_frontier usage with the existing count and filter operators. The issue asks for an example of a less-generic custom operator and reports a Stream type mismatch; done would be a documented, working pattern for defining and importing such an operator.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100