max-mapper / max-mapper/nugget
Streaming download pool for JS API
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 178
- Forks
- 29
- PR merge metrics
- No merged PRs in 30d
Description
it might make sense to do this as a separate module, and this might already exist, but I want this API:
// for example purposes assume urlStream is an object stream that emits a bunch of URLs as strings, or objects for request options
var nugget = require('nugget')
var downloader = nugget.createDownloadStream() // options could be the parallelism, and also you could pass defaults for request here as options.request or options.defaults maybe
pump(urlStream, downloader, function (err) {
if (err) throw err
console.log('done downloading')
})
Internally, createDownloadStream would start a configurable sized parallel queue (maybe powered by https://www.npmjs.com/package/run-parallel-limit). It would return a writable stream that you write urls into.
For every URL received, it should add it to the queue. It should emit events for when it starts and finishes each URL, as well as expose download progress through a static property/object somewhere on the createDownloadStream instance.
Error handling, it should only destroy the stream with an error if it's a catastrophic error. Maybe you can pass in a function that gets called with the (err, resp, body) for each request and that way you can handle the response yourself if you want?
Finally, when it downloads, it should do it like nugget/wget where it saves the resource to a file on disk. The file it saves as should be configurable in the object you write in as input. If you just write a single URL string as input, it should do with nugget does by default -- just use the http filename.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the proposed createDownloadStream() entry point and review nugget's existing download behavior, including the suggested run-parallel-limit dependency. Done means a writable stream accepts URL strings or request objects, downloads with configurable parallelism and filenames, reports progress and per-URL events, and handles errors as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100