detect length of file-like object?

Originally opened by @skshetry on 2021-03-01 08:38:51 in encode/httpx

Checklist

• There are no similar issues or pull requests for this yet.
• I discussed this idea on the community chat and feedback is positive.

Is your feature related to a problem? Please describe.

I'm using httpx to upload files to a WebDAV server, which uses PUT. The problem here is that not a lot of server has chunked uploading support for PUT.

When uploading a file-like object, request tries to detect file size to send it in Content-Length header using super_len(). But httpx uses Transfer-Encoding: chunked. So, the files end up empty or the server throws a 411 Length Required response.

The difference in behaviour with respect to requests can be demonstrated as follows:

>>> import requests, httpx
>>> from io import BytesIO

>>> requests.put("https://httpbin.org/anything", data=BytesIO(b"foobar")).request.headers
 {'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '6'}

>>> httpx.put("https://httpbin.org/anything", data=BytesIO(b"foobar")).request.headers
Headers({'host': 'httpbin.org', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.16.1', 'transfer-encoding': 'chunked'})

Describe the solution you would like.

It'd be great to have the same behaviour as requests here, i.e. trying to detect the size and then falling back to chunked encoding.

Describe alternatives you considered

As a workaround, I can use peek_filelike_length and a custom header Content-Length which is respected, that works well for me.

Additional context

We had a long discussion in https://github.com/iterative/dvc/issues/4796 regarding it. nextcloud-snap that I use locally for testing, does not have support this either due to a bug in Apache (or, some modules in it): https://github.com/nextcloud/nextcloud-snap/issues/365.

And, Sabredav (owncloud/nextcloud uses them internally) wiki has a good insight on this (though the article is >6 years old):

Finder uses Transfer-Encoding: Chunked in PUT request bodies. This is a little used HTTP feature, and therefore not implemented in a bunch of web servers. The only server I've seen so far that handles this reasonably well is Apache + mod_php. Nginx and Lighttpd respond with 411 Length Required, which is completely ignored by Finder. This was seen on nginx 0.7.63. It was recently reported that a development release (1.3.8) no longer had this issue.

When using this with Apache + FastCGI PHP completely drops the request body, so it will seem as if the PUT request was successful, but the file will end up empty.

Note that the recent servers do support chunked encoding, but we still find a lot of old servers or are using certain modules that are buggy wrt chunked encoding.

I understand that it's the server that is not http/1.1 compliant. And, if httpx does not want to support, I'm okay with the workaround as well. But I'd happy to make a PR if we decide to do so. Thanks. 🙂

Start with request body handling and httpx/_utils.py, especially peek_filelike_length, and compare the demonstrated requests behavior for BytesIO uploads. Determine how detectable file-like lengths should produce Content-Length while other bodies retain chunked encoding; done when the PUT example no longer requires a custom header and the fallback behavior remains intact.