error when trying to copy large files which were written to GCS, s3fs multi-part upload to blame?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
Version of s3fs being used (s3fs --version)
s3fs==2021.7.0
Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)
Kernel information (uname -r)
5.8.0-44-generic
GNU/Linux Distribution, if applicable (cat /etc/os-release)
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Details about issue
My app uses s3fs in python to write moderately large files (12-25 MBytes), using the S3FileSystem client
with calls to s3.open(), calls to write(), and then close() the file
The files are written to GCS, but if I try to copy them to another bucket, or within the same bucket, using 'gsutil cp', I get this error
BadRequestException: 400 Rewriting objects created via Multipart Upload is not implemented yet. As a workaround, you can use compose to overwrite the object (by specifying leela-yoyodyne-dev/cameras/M2/A1/2022-06-09/21/2022-06-09T21:39:35.062Z.obj.jsonl as both the source and output of compose) prior to rewrite.
Is there some way to avoid creating multipart files?
I create a client like this
self.s3 = S3FileSystem(
anon=False,
key=access_key_id,
secret=secret_access_key,
client_kwargs={
'endpoint_url': endpoint
})
Are there some extra options to create the S3FileSystem such that it would not use multipart uploads, to prevent this issue? Or some other workaround to get non-composite objects created ?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with Python's S3FileSystem using the reported s3fs, Ubuntu, and GCS setup, then inspect how s3.open(), write(), and close() select multipart uploads. Check the gsutil cp rewrite limitation and available S3FileSystem options. Done means identifying a supported non-multipart path or documenting a confirmed workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100