When stream=True iter_content(chunk_size=None) reads the input as a single big chunk
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
According to the documentation when stream=True iter_content(chunk_size=None) "will read data as it arrives in whatever size the chunks are received", But it actually collects all input into a single big bytes object consuming large amounts of memory and entirely defeating the purpose of iter_content().
Expected Result
iter_content(chunk_size=None) yields "data as it arrives in whatever size the chunks are received".
Actual Result
A single big chunk
Reproduction Steps
from requests import get
URL = 'https://dl.fedoraproject.org/pub/alt/iot/32/IoT/x86_64/images/Fedora-IoT-32-20200603.0.x86_64.raw.xz'
r = get(URL, stream=True)
for b in r.iter_content(chunk_size=None):
print(len(b))
prints
533830860
System Information
$ python -m requests.help
{
"chardet": {
"version": "3.0.4"
},
"cryptography": {
"version": "2.9.2"
},
"idna": {
"version": "2.9"
},
"implementation": {
"name": "CPython",
"version": "3.7.6"
},
"platform": {
"release": "4.19.104+",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "1010107f",
"version": "19.1.0"
},
"requests": {
"version": "2.23.0"
},
"system_ssl": {
"version": "1010107f"
},
"urllib3": {
"version": "1.25.9"
},
"using_pyopenssl": true
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Response.iter_content and the linked API documentation, then run the provided Python reproduction against the Binder example to observe chunk sizes and memory behavior. Done means iter_content(chunk_size=None) yields data as it arrives rather than collecting the entire response into one bytes object.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100