forwardemail / forwardemail/superagent
Piping data but only on 200 OK
- Dominant language
- JavaScript
- Stars
- 16.6k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
I want to download/upload large files with superagent. `.pipe()` conveniently streams data, but it inconveniently ignores all HTTP errors. I want to separate HTTP 404, 500, 403, etc as errors to stderr instead of writing their error pages out as if they were my content.
I'm having trouble wrapping my head around why this isn't allowed:
```
// get.js
let request = require('superagent');
var req = request.get(process.argv[2])
req.buffer(false)
req.then(res => {
console.log(res.status)
res.pipe(process.stdout)
})
.catch(err => {
console.log('error!')
console.log(err.status, err.message)
})
```
```
$ node get.js https://google.com
(node:44322) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
(Use `node --trace-deprecation ...` to show where the warning was created)
200
error!
undefined .end() was called twice. This is not supported in superagent
```
```
$ node get.js https://google.com/404
error!
404 Not Found
```
I get the exact same output if I use `req` as suggested in https://github.com/visionmedia/superagent/issues/1188 (as suggested [in the docs](https://visionmedia.github.io/superagent/#piping-data)):
```
req.buffer(false)
req.then(res => {
console.log(res.status, res.message)
- res.pipe(process.stdout)
+ req.pipe(process.stdout)
})
.catch(err => {
console.log('error!')
```
I understand that `.pipe()` and `.then()` are incompatible because `.then()` triggers the parsing system (https://github.com/visionmedia/superagent/issues/1187#issuecomment-281995106) -- but https://github.com/visionmedia/superagent/issues/1187#issuecomment-281995580 makes it sound like `.buffer(false)` should sidestep the parsing system.
I am wondering how to download data in a stream without buffering it all while also looking at the headers.
There is this comment https://github.com/visionmedia/superagent/issues/1187#issuecomment-281995580
> It is in a weird place that only your parser sees. #950
but I'm having trouble understanding that. https://github.com/visionmedia/superagent/issues/950#issuecomment-282112032 offers
> I think buffered responses should be the default, and unbuffered responses should be only opt-in.
but that doesn't explain to me how to get the body content.
---
I'm trying to achieve what I can do in `python-requests`: https://stackoverflow.com/questions/16694907/download-large-file-in-python-with-requests/16696317#16696317
```
def download_file(url):
local_filename = url.split('/')[-1]
# NOTE the stream=True parameter below
with requests.get(url, stream=True) as r:
r.raise_for_status()
with open(local_filename, 'wb') as f:
for chunk in r.iter_content(chunk_size=8192):
# If you have chunk encoded response uncomment if
# and set chunk_size parameter to None.
#if chunk:
f.write(chunk)
return local_filename
```
---
`r.raise_for_status()` checks `r.status` -- which it has because at that point it has parsed the HTTP headers -- and throws an exception on non-200s. But the rest of the data hasn't been read off the socket at that point so it won't crash on e.g. 2GB files.
--
I'm putting this out there in case you, or any user of the library, has a better clue how to do this than I've found so far. I don't expect you to go out of your way to solve my problem for me. Thanks for your work on superagent, and I hope you are somewhere with friends and family during the lockdowns.
Contributor guide
Assessment
This issue has not been assessed yet.