PUT requests for zero-byte files get 'Transfer-Encoding': 'chunked' header
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
Context: uploading files to AWS S3 via requests. Files may be empty. When attempting to upload an empty file, a response of 501: Not implemented is received. This seems to be because, when content length is 0, a Transfer-Encoding: chunked header is automatically added. S3 does not support the encoding.
Expected Result
When an empty file PUT is attempted via requests.put, a Content-Length: 0 header should be set on the request. A Transfer-Encoding: chunked header should not be set automatically.
Actual Result
When an empty file PUT is attempted via requests.put, a Transfer-Encoding: chunked header is set. No Content-Length header is set. If a Content-Length: 0 header is set explicitly, the Transfer-Encoding: chunked header is still set.
Reproduction Steps
If AWS account, bucket, credentials for principal with PutObject permissions aren't available:
import requests
file = open(<path_to_empty_file>, "rb")
session = requests.session()
link = "http://httpbin.org/put"
# using requests.put yields the same results. Using session.put here to examine request
# with S3 instead of httpbin.org, the call below results in a 501. Note the request headers
res_no_CL = session.put(link, data=file, headers={"Content-Type": "text/plain"})
print(res_no_CL.request.headers)
# with S3 instead of httpbin.org, the call below results in a 501. With httpbin.org it returns
# ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
# so can't examine headers, but I assume behaviour is consistent
res_explicit_CL = session.put(link, data=file, headers={"Content-Type": "text/plain", "Content-Length": "0"})
print(res_explicit_CL.request.headers)
If AWS prerequisites above are available:
import requests
import boto3
s3 = boto3.client('s3')
file = open(<path_to_empty_file>, "rb")
session = requests.session()
link = s3.generate_presigned_url(ClientMethod="put_object", Params={"Bucket": "<your_bucket>", "Key": "<path_in_bucket>", "ContentType": "text/plain"})
# using requests.put yields the same results. Using session.put here to examine request
# the call below results in a 501. Note the request headers
res_no_CL = session.put(link, data=file, headers={"Content-Type": "text/plain"})
print(res_no_CL.request.headers)
# the call below results in a 501. Note the request headers
res_explicit_CL = session.put(link, data=file, headers={"Content-Type": "text/plain", "Content-Length": "0"})
print(res_explicit_CL.request.headers)
System Information
$ python -m requests.help
{
"chardet": {
"version": null
},
"charset_normalizer": {
"version": "2.1.1"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "3.4"
},
"implementation": {
"name": "CPython",
"version": "3.9.5"
},
"platform": {
"release": "5.10.102.1-microsoft-standard-WSL2",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.28.1"
},
"system_ssl": {
"version": "1010106f"
},
"urllib3": {
"version": "1.26.12"
},
"using_charset_normalizer": true,
"using_pyopenssl": false
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue through the requests.put or session.put entry points with an empty file, then inspect the prepared request headers and transfer-encoding path. Done means an empty PUT sends Content-Length: 0 without Transfer-Encoding: chunked, including when Content-Length is supplied explicitly, with regression coverage for the reported cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100