Consider having bucket be a configuration parameter for `S3FileSystem`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
Background
In s3, the concept of a bucket and a key are separate. In the current design of s3fs, both are used together to define a full path. This poses a number of complications:
- When parsing
s3://bucket/path/to/file, traditional uri schemes would havebucketbe the host, and the remainder be the path. Indaskwe've ended up special casings3fsandgcfswhen handling this parsing, which is unfortunate. If the bucket was part of the filesystem, this special casing would no longer be necessary. - Buckets must be handled differently from paths. See e.g. the code for
touch(https://github.com/dask/s3fs/blob/master/s3fs/core.py#L793-L808) is split depending on if the path is just a bucket or a full key. Buckets have different semantics, when in a filesystem all objects should have the same semantics. Seeglobfor another example (https://github.com/dask/s3fs/blob/master/s3fs/core.py#L548-L549). You can't glob across buckets, because that's all ofs3. - Certain s3 operations only work on objects all under the same bucket. See e.g. https://github.com/dask/s3fs/blob/master/s3fs/core.py#L708-L737
- In use it seems many projects rely on a single bucket, so having a bucket for a filesystem object doesn't seem to be that restrictive. This is only per a small sample of users (really just people in my office), so this may not be 100% true.
A proposal for changing
Since s3fs is already in use, we can't perform a breaking change like this without some deprecation cycle. I'm not actually sure if we need to break the old behavior though, but supporting both might make the code more complicated. Proposal:
- Add a
bucket=Nonekwarg to the__init__ofS3FileSystem. The default is the current behavior. - If
bucketis specified, it becomes the bucket in all specified paths.
Since s3fs currently works with paths prefixed with s3://, I'm not sure how that should be handled. IMO s3fs shouldn't accept paths specified with s3:// (and should instead accept plain paths as if they were local paths), but that may be hard to deprecate cleanly. Perhaps if the filesystem has a bucket specified, error on paths that start with s3://.
If deprecation of the existing behavior is desired, a warning could be raised on __init__ whenever bucket is None, and eventually the existing behavior phased out.
Anyway, this is just one person's opinion. Figured it'd be worth bringing up to get some feedback.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The proposal centers on S3FileSystem.init and path handling in s3fs/core.py, including touch and glob (the issue links lines 548–549 and 793–808). First map how bucket/key parsing and same-bucket operations work, then clarify compatibility and deprecation behavior. Done requires an agreed API and implementation of that design.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100