IntelLabs / IntelLabs/vdms

S3 Object Naming May Be Excessively Nested

Open
#190 0 comments 0 reactions 0 assignees View on GitHub
Enhancement Priority: Backlog
Dominant language
C++
Stars
93
Forks
34
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

When using the S3 connector, objects are created with the full path that one would use on a local file system for storage. While this provides for an easy "caching" method for objects, its messy in practice.

The intent of these large paths for local storage is to prevent individual directories from becoming overloaded and slowing down, however this may not make sense on object storage systems, and depending on the choice of backend may even be pathological.

**Describe the solution you'd like**

Consider a different naming/nesting scheme when using remote object storage for binary data.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.