Automattic / Automattic/knox

putStream Error: read ECONNRESET

Open
#296 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
1.7k
Forks
283
PR merge metrics
No merged PRs in 30d

Description

Love knox :heart:

``` js
var http = require("http");
var https = require("https");

//use either http or https, depending on url protocol
http.get(url, function (res) {
var headers = {
'Content-Length': res.headers['content-length'],
'Content-Type': res.headers['content-type'],
'x-amz-acl': 'public-read'
};
var s3Req = s3Client.putStream(res, "/file-path", headers, function (error, res) {
if(error) {
return console.log("Error", error);
} else {
res.resume();
res.on("end", function() {
console.log("ENDED");
});
res.on("error", function() {
console.log("Error");
});
}
});
s3Req.on("error", function(error) {
console.log("s3Req Error", error);
});
});
```

I'm using this script to backfill a bunch of URLs to s3. This script works fine for uploading the first ~1,100 but I ultimately need to do about ~80,000

Once it uploads the first 1,000 it throws this error:

``` js
events.js:85
throw er; // Unhandled 'error' event
^
Error: read ECONNRESET
at exports._errnoException (util.js:746:11)
at TCP.onread (net.js:550:26)
```

I'm fairly new to node so please forgive me if I'm missing something obvious here.

I found these issues here that I _think_ are related https://github.com/Automattic/knox/issues/259 https://github.com/Automattic/knox/issues/198 but after struggling with it and trying different things I didn't come to a solution.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the putStream call and its request error handling, then review the related issues #259 and #198. Reproduce the failure by running the provided backfill pattern at the reported scale and trace the unhandled error event. Done means the upload process handles connection resets without crashing during the full batch.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, javascript, node.js
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.