Automatically log requests and response codes

The project I'm currently working on wraps treq so that it logs requests made.

Now that the new twisted logger is in place, it seems to be a useful thing to automatically include in treq as well, logging at either the INFO or DEBUG level.

I was thinking of implementing this in the following manner:

1. Generate a request ID per request, and log the request ID, the method, the url, the headers, the data (if included), and whether or not files were included, before actually issuing the request.
2. Once a response is received, log the request ID, the headers, and the response code.
3. If an error occurs while making the request, log the request ID and the error (not at the ERROR level - just the same log level everything else is logged).

But came across the following caveats:

1. I'd like to be able to pass in extra parameters to include in the log event, which may be useful in tracing why a request was made. For instance, if this request was made in response to some event, then include the event ID, which we may not necessarily want to include in the request headers.
2. Logging the request body, or even the headers or URL, might be a security risk, since maybe the request body contains password information if you are making an auth request. Maybe the headers contain API keys. Maybe the URL contains API keys or capability hooks you don't want logged.

I'm not sure how to solve the first - it seems like it'd require treq either taking a logging callable with kwargs already bound to it, or taking an extra parameter containing extra logging kwargs.

And as for the second - it seems like there need to be more fine-grained filters on what kind of information can be logged? Maybe not logging request bodies or headers at all, or only logging some? Maybe not providing logging at all for some URLs... Not sure what this should look like.

Review the existing request entry points in treq alongside otter/util/logging_treq.py and the new Twisted logger. Before implementation, resolve the API for extra logging fields and the security policy for URLs, headers, bodies, and selected URLs; done means request, response, and error events are logged without exposing sensitive data.