caolan / caolan/highland

Dynamic piping

Open
#316 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3.4k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Is it possible to resolve the problem below with highland.js?

Consider a problem:
- split file by lines
- write lines to a result file
- if a result file exceeds some size create a new result file

For example, if I have a file which weights 4gb and split size is equal 1gb. The result is four files weights 1gb.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Highland's stream and piping APIs, focusing on how streams can dynamically route data to output files. Define the expected behavior for splitting a large input by line and rotating output files at a size limit. Done means the behavior is implemented and verified for the four-file, 4GB-at-1GB-per-file example.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.