Join small partitions when manual pivot_keys are used
Open
mapreduce
- Dominant language
- C++
- Stars
- 2.2k
- Forks
- 219
- PR merge metrics
- No merged PRs in 30d
Description
I have an MR for which I known all pivot_keys (all unique tuples for reduce_by) and mapper that produces sorted output within pivot_keys[i; i + 1] range.
I want to produce sorted table by this MR operation.
But right now I’m suffering because I get too many reduce jobs (one job per any pivot_key) and some of these jobs have very small input.
With CRI runtime it works very badly because when cluster is overloaded containers spawns in 30-50 seconds and job time is only 1-20 seconds:
Contributor guide
Assessment
This issue has not been assessed yet.