fsspec / fsspec/s3fs

The acl parameter of open() is ignored when uploading an object smaller than blocksize

Open
#392 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted
Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

What happened:

with s3.open(s3_uri, 'wb', acl='bucket-owner-full-control') as f:
    f.write(b'x' * (5 * 2 ** 20 - 1))

does not obey the acl parameter (the resulting owner is incorrect), whereas

with s3.open(s3_uri, 'wb', acl='bucket-owner-full-control') as f:
    f.write(b'x' * (5 * 2 ** 20))

does.

What you expected to happen:

For the acl parameter to work regardless of object size.

Minimal Complete Verifiable Example:

with s3.open(s3_uri, 'wb', acl='bucket-owner-full-control') as f:
    f.write(b'x' * (5 * 2 ** 20 - 1))

Anything else we need to know?:

I tested this with s3fs 0.4.2 because I can't currently install 0.5.x (dependency issues), but I suspect it is broken even on current master, because it's probably this if-statement causing the use of the single-part code path that does not use self.acl the way the multi-part code path does.

Environment:

  • Dask version: 2.11.0
  • Python version: 3.6.10
  • Operating System: Linux (on SageMaker)
  • Install method (conda, pip, source): It's SageMaker (conda) with a bunch of packages installed on top with pip, I'm not sure what exactly gets in as dependencies (but one is explicit s3fs<0.5). If this turns out to matter, I'll look into getting a cleaner install, but it probably doesn't.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in s3fs/core.py at the if-statement around line 1415 and compare the single-part path around line 1535 with the multipart path around line 1422. Reproduce the reported behavior with objects just below and at 5 MiB, then verify that the acl parameter is honored for both paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.