drivendataorg / drivendataorg/cloudpathlib
Handle `mkdir` for cloud providers that support creating directories
- Dominant language
- Python
- Stars
- 628
- Forks
- 88
- Avg merge
- 17h 28m
- Merged PRs (30d)
- 2
Description
S3 has an interesting situation with folders.
Like other object stores like Azure, it has a flat structure, and when you upload a file to `a/b/c.txt` for example, it creates an object literally named `a/b/c.txt`. The directories `a` and `b` aren't real and don't exist. The web console has special behavior to fake those as folders in the UI. When you delete `c.txt`, `a` and `b` will automatically be gone.
However, S3 does have another mechanism that lets you have folders. There is a "Create Folders" button in the web console, which lets you make a folders that exist even while empty. These turn out to actually be dummy files with a trailing slash. So if you create `a/`, there is actually an object in your bucket `a/` which is not actually a folder, but the S3 console will treat it like a folder for the UI. You can equivalently upload a file to `a/` and it will do the same thing.
We need to think through the implications of this and what cloudpathlib should support.
- pathlib `PurePosixPath` strips trailing `/` on instantiation, so this is something that doesn't map to a representation cleanly through `PurePosixPath`
- ~As a result, it's not currently possible to create an `S3Path` object that points to an S3 folder.~ EDIT: This was incorrect. See discussion in #190. This is possible because the string representation of the input URI is the main basis for a CloudPath object, not a `PurePosixPath`.
- Additionally, the `S3Path.mkdir` method is not implemented.
Contributor guide
Assessment
This issue has not been assessed yet.