aio-libs / aio-libs/aiohttp

Client support for compressed transfer encoding

Open
#4,435 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
16.5k
Forks
2.4k
Avg merge
17h 22m
Merged PRs (30d)
212

Description

## Long story short
aiohttp does not support any transfer encoding other than chunking. Responses from servers using e.g. `Transfer-Encoding: gzip` result in the compressed payload. Worse yet, those applying both compression and chunking result in the raw payload with chunking still intact (but only when using the C parser!).

## Expected behaviour
Decompressed payload.

## Actual behaviour
Compressed or compressed + chunked payload.

## Steps to reproduce
A simple server and client illustrating this issue can be found in [this gist](https://gist.github.com/JustAnotherArchivist/ba23cb434810254a7d55ec893e45e1d4). I wrote a single test client and server for both this issue and #4436; the first line is irrelevant to this issue and not included in the output below.

Expected output:
```
> python3 client.py | tail -n+2
b'Test'
b'Test'
```

Actual output:
```
> python3 client.py | tail -n+2
b'\x1f\x8b\x08\x00\x82Y\xf0]\x02\xff\x0bI-.\x01\x002\xd1Mx\x04\x00\x00\x00'
b'5\r\n\x1f\x8b\x08\x00\x82\r\n13\r\nY\xf0]\x02\xff\x0bI-.\x01\x002\xd1Mx\x04\x00\x00\x00\r\n0\r\n\r\n'
```

However, when running it with the pure-Python parser, the chunked TE gets handled in the second case, and the output is:
```
> AIOHTTP_NO_EXTENSIONS=1 python3 client.py | tail -n+2
b'\x1f\x8b\x08\x00|\\\xf0]\x02\xff\x0bI-.\x01\x002\xd1Mx\x04\x00\x00\x00'
b'\x1f\x8b\x08\x00|\\\xf0]\x02\xff\x0bI-.\x01\x002\xd1Mx\x04\x00\x00\x00'
```

## Your environment
I tested this with aiohttp 2.3.10 and Python 3.6.9 on Debian, but based on the current aiohttp code, the behaviour should still be the same on the current versions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.