TimelyDataflow / TimelyDataflow/differential-dataflow
Is this the most efficient way to do a left join?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
I want to left-join fairly regularly in my data flow, and have written this helper:
fn left_join<G: Scope, K: Data, V: Data, R: Diff, V2: Data>(a: &Collection<G, (K, V), R>, b: &Collection<G, (K, V2), R>) -> Collection<G, (V,Option<V2>), <R as Mul<R>>::Output>
where R: Mul<R, Output = R>, K: Data + Hashable, G::Timestamp: Lattice + Ord {
let with = a.join_map(b, |_, v, v2| (v.clone(), Some(v2.clone())));
let without = a.antijoin(&b.map(|(k, _)| k)).map(|(_, v)| (v, None));
with.concat(&without)
}
FWIW, I often combine this with a
.group(move |_key, input, output| {
let out = input.into_iter().filter_map(|&(opt, _)| opt.as_ref()).cloned().collect::<Vec<_>>();
output.push((out, 1));
})
which gives me a Collection<G, (V, Vec<V2>), R>.
Is this reasonably efficient? (Also curious if we could get something similar in the standard library?)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the shown left_join helper and its join_map, antijoin, concat, and group calls; compare their behavior and costs before considering a library API. The issue names no files or tests, so done is not defined beyond establishing an efficient left join and deciding whether a standard helper should exist.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100