binpash / binpash/pash

Load balancing debugging info

Open
#728 0 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Shell
Stars
601
Forks
52
PR merge metrics
No merged PRs in 30d

Description

PaSh currently does not do any rebalancing of outputs between stages of the pipeline with the same width. This could end up in pathological scenarios, e.g., when the input of a program `cat IN | cmd1 | cmd2` is one line, cmd1 and cmd2 are both stateless, and cmd1 creates a bunch of lines that can then be processed by cmd2 in parallel, PaSh will not get any parallelism in this case.

To help identify such pathological scenarios it would be great if we could add a flag that adds logging nodes in parts of the dataflow that print how many lines and bytes go through them.

The steps to get this done would be to:
1. Implement a command that simply forwards its input to its output (no buffering like dgsh-tee), but also measures and prints the number of bytes and lines at the end.
2. Add this command after each stage of the dataflow graph to get the load for each different parallel line of the graph
3. (optional) Create a simple post-processing tool that can present the output in a nice way (relative loads or even in a plot, see for example `--graphviz` option in current PaSh).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how PaSh builds its dataflow graph and how the existing --graphviz option exposes that graph. Implement the forwarding measurement command, then determine where to place it after each stage so it reports lines and bytes per parallel path; done means the flag produces usable load information without changing pipeline output.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
cli, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.