Response.content iterates in needlessly small chunks

Response.content iterates over the response data in chunks of 10240 bytes. The number 10240 was set in commit 62d2ea8.

After tracing the source code of urllib3 and httplib, I can’t see a reason for this behavior. It all ultimately goes through httplib’s HTTPResponse.readinto, which automatically limits the read size according to Content-Length or the chunked framing.

Therefore, it seems that, if you simply set CONTENT_CHUNK_SIZE to a much larger number (like 10240000), nothing should change, except Response.content will become more efficient on large responses.

Update: it seems like httplib allocates a buffer of the requested size (to be read into), so simply setting CONTENT_CHUNK_SIZE to a large value will cause large chunks of memory to be allocated, which is probably a bad idea.

This is not a problem for me and I have not researched it thoroughly. I’m filing this issue after investigating a Stack Overflow question where this caused an unexpected slowdown for the poster, and a subsequent IRC exchange with @Lukasa. Feel free to do (or not do) whatever you think is right here.

Start at requests/models.py around the Response.content implementation and trace CONTENT_CHUNK_SIZE through urllib3 and http.client's HTTPResponse.readinto. Measure large-response performance and buffer allocation before choosing a change. Done means the content path avoids needless slowdown without unacceptable memory use, with regression coverage for the behavior.