fsspec / fsspec/s3fs

requests_toolbelt.MultipartEncoder fails to encode S3File objects due to missing `len` property

Open
#933 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

Hello,

I'm unable to use a S3File object together with requests_toolbelt.MultipartEncoder (https://toolbelt.readthedocs.io/en/latest/user.html). The MultipartEncoder expects the File object to contain len property to calculate its size and fails with the following:

File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 125, in __init__
    self._prepare_parts()
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 246, in _prepare_parts
    self.parts = [Part.from_field(f, enc) for f in self._iter_fields()]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 246, in <listcomp>
    self.parts = [Part.from_field(f, enc) for f in self._iter_fields()]
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 495, in from_field
    return cls(headers, body)
           ^^^^^^^^^^^^^^^^^^
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 488, in __init__
    self.len = len(self.headers) + total_len(self.body)
                                   ^^^^^^^^^^^^^^^^^^^^
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 432, in total_len
    if hasattr(o, 'len'):
       ^^^^^^^^^^^^^^^^^
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 573, in len
    return total_len(self.fd) - self.fd.tell()
           ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

It works with a custom adapter like this:

class S3FileWithLenAdapter:
    def __init__(self, s3_file):
        self._s3_file = s3_file

    @property
    def len(self):
        return self._s3_file.size

    def __getattr__(self, name):
        return getattr(self._s3_file, name)

Could you consider adding len property to S3File objects?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the S3File implementation and reviewing its existing size handling, then compare it with requests_toolbelt/multipart/encoder.py, especially total_len and the S3File-related len access shown in the traceback. Done means MultipartEncoder can consume an S3File without the reported TypeError; add or update focused tests if the repository provides them.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.