S3 Object Naming May Be Excessively Nested
Open
Enhancement
Priority: Backlog
- Dominant language
- C++
- Stars
- 93
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
When using the S3 connector, objects are created with the full path that one would use on a local file system for storage. While this provides for an easy "caching" method for objects, its messy in practice.
The intent of these large paths for local storage is to prevent individual directories from becoming overloaded and slowing down, however this may not make sense on object storage systems, and depending on the choice of backend may even be pathological.
**Describe the solution you'd like**
Consider a different naming/nesting scheme when using remote object storage for binary data.
Contributor guide
Assessment
This issue has not been assessed yet.