Dynamic piping
Open
- Dominant language
- JavaScript
- Stars
- 3.4k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Is it possible to resolve the problem below with highland.js?
Consider a problem:
- split file by lines
- write lines to a result file
- if a result file exceeds some size create a new result file
For example, if I have a file which weights 4gb and split size is equal 1gb. The result is four files weights 1gb.
Contributor guide
Research direction
Start by reviewing Highland's stream and piping APIs, focusing on how streams can dynamically route data to output files. Define the expected behavior for splitting a large input by line and rotating output files at a size limit. Done means the behavior is implemented and verified for the four-file, 4GB-at-1GB-per-file example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100