putStream Error: read ECONNRESET
- Dominant language
- JavaScript
- Stars
- 1.7k
- Forks
- 283
- PR merge metrics
- No merged PRs in 30d
Description
Love knox :heart:
``` js
var http = require("http");
var https = require("https");
//use either http or https, depending on url protocol
http.get(url, function (res) {
var headers = {
'Content-Length': res.headers['content-length'],
'Content-Type': res.headers['content-type'],
'x-amz-acl': 'public-read'
};
var s3Req = s3Client.putStream(res, "/file-path", headers, function (error, res) {
if(error) {
return console.log("Error", error);
} else {
res.resume();
res.on("end", function() {
console.log("ENDED");
});
res.on("error", function() {
console.log("Error");
});
}
});
s3Req.on("error", function(error) {
console.log("s3Req Error", error);
});
});
```
I'm using this script to backfill a bunch of URLs to s3. This script works fine for uploading the first ~1,100 but I ultimately need to do about ~80,000
Once it uploads the first 1,000 it throws this error:
``` js
events.js:85
throw er; // Unhandled 'error' event
^
Error: read ECONNRESET
at exports._errnoException (util.js:746:11)
at TCP.onread (net.js:550:26)
```
I'm fairly new to node so please forgive me if I'm missing something obvious here.
I found these issues here that I _think_ are related https://github.com/Automattic/knox/issues/259 https://github.com/Automattic/knox/issues/198 but after struggling with it and trying different things I didn't come to a solution.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the putStream call and its request error handling, then review the related issues #259 and #198. Reproduce the failure by running the provided backfill pattern at the reported scale and trace the unhandled error event. Done means the upload process handles connection resets without crashing during the full batch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, javascript, node.js
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100