Unexplainable size limit for switching to multipart upload
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
This isn't an observed bug. However, when looking at the code, I found it odd that _put_file and _pipe_file switch to multipart at 5 MB but use 50 MB chunks by default. In my opinion, the limit should be the specified chunk size.
https://github.com/dask/s3fs/blob/38bceeed4bc7de6db6dc06ce67149aa9a8f49699/s3fs/core.py#L789-L796
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in s3fs/core.py at lines 789-796, where _put_file and _pipe_file choose when to switch to multipart upload. Compare that threshold with the specified default chunk size, then verify that the switch uses the intended limit and that the resulting upload behavior matches the issue's request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100