apache / apache/arrow-rs-object-store
Allow ignoring checks for empty segments in paths (double slash)
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I was using obstore library which uses this package and encountered an issue with weird paths in our object storage (azure) ( https://github.com/developmentseed/obstore/issues/429 for reference ). We had some paths that contained double slash which is allowed by azure service. These kinds of paths throw an exception when listing contents of a container.
**Describe the solution you'd like**
It would be nice to be able to catch this exception or just ignore this and print warning as right now it stops the iterator from yielding paths. Maybe a flag? Or to have implementations for specific services allow these kinds of paths if the service allows them (like azure).
**Describe alternatives you've considered**
Unfortunately I can't always fix these paths as its coming from other teams so have to deal with messy data lake.
**Additional context**
_(issue linked above)_
Contributor guide
Research direction
Start by tracing path validation during container listing and review the linked obstore issue for the double-slash behavior. Clarify whether the completed change should catch the exception, warn and continue, or provide a service-specific or configurable allowance; done means supported double-slash paths no longer stop iteration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100