Incorrect Content-Length header with StringIO body
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
When requests is used with an io.StringIO as the data type, and the body contains characters whose utf-8 encoding is multiple bytes, the Content-Length header is set incorrectly.
Looking at the implementation of super_len, it appears that io.StringIO has its length measured using seek and tell.
It has been implemented that way since June 2016 (af7729f64a97ab35e83a1a7971781e69d124d99e).
It looks like this was fixed for str inputs in #6586 in 2023 but was never fixed for io.StringIO
I am happy to send a PR if the implementation is straightforward.
Off the top of my head I don't know how to count the bytes in a utf-8 encoded StringIO without copying, and previous PRs have tried to avoid a copy in super_len
Expected Result
Content-Length should match the number of bytes sent when using io.StringIO
Actual Result
Content-Length is the length of the string, not the bytes sent.
Reproduction Steps
Run the following script, which shows the problem in detail
import io
import requests
from urllib3.connection import HTTPConnection
from requests.utils import super_len
from requests.models import PreparedRequest
# A string that is 1 character but 4 bytes in UTF-8.
# requests will always send 4 bytes, but the Content-Length header depends on the type passed as `data`.
value = "💩"
body_types = [{
"name": "str",
"value": value,
}, {
"name": "bytes",
"value": value.encode("utf-8"),
}, {
"name": "io.BytesIO",
"value": io.BytesIO(value.encode("utf-8")),
},
{
"name": "io.StringIO",
"value": io.StringIO(value),
}]
print("## Super Len")
for body_type in body_types:
print("Body Type:", body_type["name"])
print("Super Len:", super_len(body_type["value"]))
p = PreparedRequest()
p.prepare(
method="POST",
url="http://example.com",
data=body_type["value"],
)
print("Prepared Headers:", p.headers)
# Monkey patch to print the data sent.
old_send = HTTPConnection.send
def new_send(self, data):
print("Sending Data:", data)
old_send(self, data)
HTTPConnection.send = new_send
print("## Requests")
for body_type in body_types:
r = requests.post(
"http://example.com",
data=body_type["value"],
)
Here is my output:
## Super Len
Body Type: str
Super Len: 4
Prepared Headers: {'Content-Length': '4'}
Body Type: bytes
Super Len: 4
Prepared Headers: {'Content-Length': '4'}
Body Type: io.BytesIO
Super Len: 4
Prepared Headers: {'Content-Length': '4'}
Body Type: io.StringIO
Super Len: 1
Prepared Headers: {'Content-Length': '1'}
## Requests
Sending Data: b'POST / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 4\r\n\r\n'
Sending Data: b'\xf0\x9f\x92\xa9'
Sending Data: b'POST / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 4\r\n\r\n'
Sending Data: b'\xf0\x9f\x92\xa9'
Sending Data: b'POST / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 4\r\n\r\n'
Sending Data: b'\xf0\x9f\x92\xa9'
Sending Data: b'POST / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 1\r\n\r\n'
Sending Data: b'\xf0\x9f\x92\xa9'
Notice that the request is always the same, except that the Content-Length header is 1 instead of 4 when io.StringIO is used.
System Information
{
"chardet": {
"version": "5.2.0"
},
"charset_normalizer": {
"version": "3.4.1"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "3.10"
},
"implementation": {
"name": "CPython",
"version": "3.12.9"
},
"platform": {
"release": "22.6.0",
"system": "Darwin"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.32.3"
},
"system_ssl": {
"version": "30400010"
},
"urllib3": {
"version": "2.3.0"
},
"using_charset_normalizer": false,
"using_pyopenssl": false
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/requests/utils.py at super_len and run the reproduction script from the issue with io.StringIO containing a multibyte UTF-8 character. Trace how the body is measured versus sent, then add regression coverage for the mismatch. Done means Content-Length equals the number of bytes sent for StringIO bodies without breaking the existing str, bytes, and BytesIO cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100