apache / apache/arrow-rs-object-store

Allow ignoring checks for empty segments in paths (double slash)

Open
#341 4 comments 1 reaction 0 assignees View on GitHub
enhancement good first issue help wanted
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I was using obstore library which uses this package and encountered an issue with weird paths in our object storage (azure) ( https://github.com/developmentseed/obstore/issues/429 for reference ). We had some paths that contained double slash which is allowed by azure service. These kinds of paths throw an exception when listing contents of a container.

**Describe the solution you'd like**
It would be nice to be able to catch this exception or just ignore this and print warning as right now it stops the iterator from yielding paths. Maybe a flag? Or to have implementations for specific services allow these kinds of paths if the service allows them (like azure).

**Describe alternatives you've considered**
Unfortunately I can't always fix these paths as its coming from other teams so have to deal with messy data lake.

**Additional context**
_(issue linked above)_

Contributor guide

Open the contributing guide

Research direction

Start by tracing path validation during container listing and review the linked obstore issue for the double-slash behavior. Clarify whether the completed change should catch the exception, warn and continue, or provide a service-specific or configurable allowance; done means supported double-slash paths no longer stop iteration.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, rust
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.