Automattic / Automattic/knox

putStream with transfer-encoding chunked

Open
#322 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
1.7k
Forks
283
PR merge metrics
No merged PRs in 30d

Description

I'm trying to stream a request response straight into S3, but I'm running into issues because the response is chunked and doesnt have a content-length:

```
headers:
{ 'content-type': 'image/jpeg',
expires: 'Fri, 02 Jun 2017 22:05:16 GMT',
date: 'Fri, 02 Jun 2017 22:05:16 GMT',
'cache-control': 'private, max-age=0, must-revalidate, no-transform',
vary: 'Accept, X-GData-Authorization, GData-Version',
'gdata-version': '1.0',
'transfer-encoding': 'chunked',
'x-content-type-options': 'nosniff',
'x-frame-options': 'SAMEORIGIN',
'x-xss-protection': '1; mode=block',
server: 'GSE',
'alt-svc': 'quic=":443"; ma=2592000; v="38,37,36,35"',
connection: 'close' },
```

Here's what I've got going (using the `request` package):

```
request.get(url)
.on("response", (res) => {
const path = randomID()
const req = client.putStream(res, path, res.headers, function (err, res) {
if (err) {
reject(err)
}
else {
resolve(req.url)
}
})
})
```

I see you talked about it here and added that null check https://github.com/Automattic/knox/issues/92 but I got it working simply by removing that check...

@TooTallNate @rauchg

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the request.get and client.putStream flow shown in the issue with a chunked response lacking content-length. Read the existing putStream handling and the behavior discussed in issue #92; done means the response can be streamed to S3 successfully without a content-length header.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, javascript
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.