forwardemail / forwardemail/superagent

Uploading a large file using stream.pipe(req) does not work

Open
#1,654 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
16.6k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Uploading files larger than the read stream's highWatermark does not work.

This is tested with node 16.13.1

Sample server:
```
const express = require('express'),
fs = require('fs'),
http = require('http');

function upload(req, res) {
const out = fs.createWriteStream('/tmp/out');
req.pipe(out).on('finish', function () {
console.log('upload finished');
res.status(200).send({});
});
}

const app = express();
app.post('/upload', upload);

http.createServer({}, app).listen(3000, function () {
console.log('listening on port 3000');
});
```

You can test the server itself works using `curl -F file=@largefile.txt -X POST http://localhost:3000/upload`

Superagent upload code:
```
'use strict';

const fs = require('fs'),
superagent = require('superagent');

const readStream = fs.createReadStream('./largefile.txt');
const request = superagent.post('http://localhost:3000/upload')
request.on('response', function (response) {
console.log('got response', response.status);
});
readStream.pipe(request);
```

The code above will just "hang" if largefile.txt is more than twice the highWaterMark. For fs, the default water mark is 64KB. So, if the file is say 200KB, the upload code will hang.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.