Chunked responses in aiohttp result in an attempt to allocate a 4 gigabyte buffer… and maybe it doesn't actually work anyway?

Commit b95ca24 replaced calls to asyncio.StreamReader.read with calls to asyncio.StreamReader.readexactly. One important distinction between these two methods is that read allows a value of -1 to be passed, which results in it reading until EOF is reached, whereas readexactly requires (as its name implies) an exact size. This change breaks "chunked" transfers, as they don't supply a "Content-Length" header. When that header isn't present, aiohttp defaults to passing -1 to the read method, resulting in it attempting to allocate a buffer of 4,294,967,126 bytes.

Interestingly, ChunkedClientResponse.read correctly decodes the content length from the first line of the response data (which is consistent with the HTTP 1.1 spec), but ignores that value in favor of the -1 passed from ClientResponse.text or ClientResponse.json. This is handled in ClientResponse.read by checking specifically for a value of -1 and using the asyncio.StreamReader.read method, but not in ChunkedClientResponse.read.

Upon further inspection…

After looking closer at ChunkedClientResponse and the HTTP 1.1 spec, I think it just doesn't work correctly at all. A chunk-encoded response can require multiple reads from the response stream, each of a length determined by a value encoded in its first line. The way the class is written assumes that the first chunk's size is the size of the full data (chunked encoding is specifically intended for situations where the full size isn't known when the transfer starts) and there is only a single chunk.

So hey, I'm going to rewrite it and do a PR.

Start with ChunkedClientResponse and compare its handling with ClientResponse.read, including the text and json entry points. Review how asyncio.StreamReader.read and readexactly are used, then verify that chunked responses can consume multiple chunks without attempting an enormous allocation. Done means chunked transfers are decoded correctly and the reported failure no longer occurs.