TimelyDataflow / TimelyDataflow/timely-dataflow

closure ergonomics with custom operators

Open
#161 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Rust
Stars
3.6k
Forks
293
Avg merge
14h 46m
Merged PRs (30d)
4

Description

I'm working on a timely program that will involve ~5 custom operators, each of which is fairly involved. Ideally I'd like to import those custom operators into main.rs for readability's sake. Something like:

input.to_stream(scope)
    .map(..)
    .custom_operator1()
    .filter(..)
    .custom_operator2()
{etc}  

where the custom operators are unary_frontiers with complicated closures.

I tried to write my own traits by copying the type signatures in operators like count and filter, but I think the issue is my custom operators are not as generic as those operators and I'm getting tripped up over the type signatures. My operators assume that each record is of the form ((id1, id2, id3, id4), Vec) so I end up getting errors like

  --> src/lib.rs:182:13
    |
169 |           fn location_aggregation(&self) -> Stream<S, D> {
    |                                             ------------ expected `operations::timely::dataflow::Stream<S, D>` because of return type
...
182 | /             self.unary_frontier(
183 | |                         exchange,
184 | |                         "agg locs",
185 | |                         |_capability| {
...   |
309 | |                             }
310 | |                         })
    | |__________________________^ expected type parameter, found tuple
    |
    = note: expected type `operations::timely::dataflow::Stream<_, D>`
               found type `operations::timely::dataflow::Stream<_, ((isize, _, _, _), std::vec::Vec<f64>)>`

I feel like this problem has more to do with my lack of rust experience than timely, so I'm not sure this is ticket-worthy. Feel free to close, if so. But do you have an example of a custom operator definition that isn't as generic as count et al?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the custom operator code in src/lib.rs and compare its unary_frontier usage with the existing count and filter operators. The issue asks for an example of a less-generic custom operator and reports a Stream type mismatch; done would be a documented, working pattern for defining and importing such an operator.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.