drivendataorg / drivendataorg/cloudpathlib

Handle `mkdir` for cloud providers that support creating directories

Open
#51 9 comments 1 reaction 0 assignees View on GitHub
design decision help wanted S3
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

S3 has an interesting situation with folders.

Like other object stores like Azure, it has a flat structure, and when you upload a file to `a/b/c.txt` for example, it creates an object literally named `a/b/c.txt`. The directories `a` and `b` aren't real and don't exist. The web console has special behavior to fake those as folders in the UI. When you delete `c.txt`, `a` and `b` will automatically be gone.

However, S3 does have another mechanism that lets you have folders. There is a "Create Folders" button in the web console, which lets you make a folders that exist even while empty. These turn out to actually be dummy files with a trailing slash. So if you create `a/`, there is actually an object in your bucket `a/` which is not actually a folder, but the S3 console will treat it like a folder for the UI. You can equivalently upload a file to `a/` and it will do the same thing.

We need to think through the implications of this and what cloudpathlib should support.
- pathlib `PurePosixPath` strips trailing `/` on instantiation, so this is something that doesn't map to a representation cleanly through `PurePosixPath`
- ~As a result, it's not currently possible to create an `S3Path` object that points to an S3 folder.~ EDIT: This was incorrect. See discussion in #190. This is possible because the string representation of the input URI is the main basis for a CloudPath object, not a `PurePosixPath`.
- Additionally, the `S3Path.mkdir` method is not implemented.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.