ytsaurus / ytsaurus/ytsaurus

Join small partitions when manual pivot_keys are used

Open
#279 2 comments 0 reactions 1 assignee Claimed by @sergey-v-galtsev View on GitHub
mapreduce
Dominant language
C++
Stars
2.2k
Forks
219
PR merge metrics
No merged PRs in 30d

Description

I have an MR for which I known all pivot_keys (all unique tuples for reduce_by) and mapper that produces sorted output within pivot_keys[i; i + 1] range.
I want to produce sorted table by this MR operation.

But right now I’m suffering because I get too many reduce jobs (one job per any pivot_key) and some of these jobs have very small input.

With CRI runtime it works very badly because when cluster is overloaded containers spawns in 30-50 seconds and job time is only 1-20 seconds:

Screenshot 2023-12-21 at 14 07 11

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.