developmentseed / developmentseed/multistore

List of a prefix 503s when a single object key contains an empty path segment (//)

Open
#116 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19
Forks
1
PR merge metrics
No merged PRs in 30d

Description

### Summary

When a ListObjectsV2 page returned by the backend contains a single object whose key has an **empty path segment** (a `//`), `multistore` fails the **entire** list request with `503 ServiceUnavailable` instead of listing the other (valid) keys. One malformed object key takes out the whole prefix.

`object_store`'s strict `Path` parser rejects empty segments, and multistore propagates that as a fatal backend error for the page rather than skipping the offending key.

### Observed error

Proxy log:

```
[WARN] multistore::proxy: request failed {
error=backend error: Encountered object with invalid path:
Path "raw//" contained empty path segment,
status=503, s3_code=ServiceUnavailable
}
```

Client just sees the generic:

```
An error occurred (ServiceUnavailable) when calling the ListObjectsV2 operation
(reached max retries: 2): Service unavailable
```

### Reproduction

An object genuinely exists in S3 with a `//` in its key (double slash after `raw/`):

```
s3://.../raw//b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DELAYED-2045.001/atm/proc/tseries/day_1/....nc
```

`aws s3 ls --no-sign-request` against the raw bucket lists it fine (S3 keys are opaque byte strings and `//` is legal). Listing the same prefix through multistore 503s.

Only prefixes that *contain* such a key fail; sibling prefixes list normally.

### Impact

A single stray upload with a double slash (or a zero-byte folder-marker object ending in `//`) makes an entire prefix unlistable through the gateway, with an error that gives the operator no hint about which object or why. S3 itself accepts and serves these keys, so the gateway is stricter than the backend it fronts.

### Suggested fix

On list, don't let one unparseable key fail the page. Options:

- **Skip** keys that fail strict `Path::parse`, ideally logging the raw key at `warn` so operators can find and fix it; or
- Parse list keys leniently (`Path::from` / `from_url_path`, which tolerate empty segments) instead of strict `parse`.

At minimum, surface the offending raw key in the error so it's diagnosable without correlating backend logs.

### Environment

- multistore 0.6.3 (features: `azure`), via `multistore-cf-workers` on Cloudflare Workers
- Backend: S3

Contributor guide

Open the contributing guide

Research direction

Start by tracing ListObjectsV2 page handling to the strict Path::parse call described in the issue, then inspect existing list-related tests. Reproduce with a key containing // and verify that one invalid key no longer causes the whole prefix listing to return 503, while the offending key is either skipped with a warning or handled leniently.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.