Show response body when debug logging enabled

Suppose I'm getting an HTTP error from the server DAV endpoints when running vdirsyncer. I then add -vdebug as suggested by the error message from vdirsyncer, expecting to see information would tell me the reason for the failure.

Unfortunately, while vdirsyncer logs the request headers and body, as well as the response status code and headers, it does not log the actual response body from the server, which is where the actual error would most likely be found. Instead, it logs this:

debug: <StreamReader 339 bytes eof>

The log in question is here: https://github.com/pimutils/vdirsyncer/blob/388c16f188a43e3253c6aa56748e035d3159c4e4/vdirsyncer/http.py#L224

I'm not sure how a user is expected to find the error message, except by repeating the HTTP request themselves and seeing what the server returns.

I thought it might be easy to contribute an improvement for this (log the response body on -vdebug), but it turns out to be a bit complicated. Firstly, if you just read the body:

logger.debug(await response.content.read())

Then it consumes the body and it can no longer be read by vdirsyncer subsequently. It would be necessary to make a copy of the response object, but I could not find any API method for doing this exposed by aiohttp or requests.

Next, I noticed the comment:

https://github.com/pimutils/vdirsyncer/blob/388c16f188a43e3253c6aa56748e035d3159c4e4/vdirsyncer/http.py#L178-L179

I implemented it, following https://docs.aiohttp.org/en/stable/client_advanced.html#aiohttp-client-tracing and https://docs.aiohttp.org/en/stable/tracing_reference.html#aiohttp.TraceConfig.on_response_chunk_received, and added to vdirsyncer.http:

async def trace_response_chunk_received(session, trace_config_ctx, params):
    logger.debug(params.chunk)

trace_config = aiohttp.TraceConfig()
trace_config.on_response_chunk_received.append(trace_response_chunk_received)

I passed trace_configs=[http.trace_config] to the instantiations of ClientSession in other modules. However, this did not work. I discovered that on_response_chunk_received does not function as documented, as mentioned in https://github.com/aio-libs/aiohttp/issues/5324.

At this point I decided to file this issue with my findings.

Start in vdirsyncer/http.py around the response logging at line 224 and the tracing comment at lines 178-179, then inspect the ClientSession instantiations in the other modules. Review the linked aiohttp tracing documentation and issue; done means debug logging exposes the server response body without preventing vdirsyncer from reading it afterward.