developmentseed / developmentseed/obspec-utils

resolve() truncates object keys containing '#', '?', or ';' (breaks special-char keys and presigned URLs)

Open Beginner friendly
#85 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
12
Forks
3
PR merge metrics
No merged PRs in 30d

Description

## Summary

`ObjectStoreRegistry.resolve()` derives the object key from `urlparse(url).path`, which silently discards anything `urlparse` treats as URL **params (`;`)**, **query (`?`)**, or **fragment (`#`)**. As a result the resolved key is truncated whenever a URL contains one of those characters.

This bites in two distinct, real situations:

### Motivation 1 — object keys that contain `#` / `?` / `;`

These characters are valid in S3/GCS/Azure object keys. Concrete example: the [Cell Painting Gallery](https://registry.opendata.aws/cellpainting-gallery/) (AWS Open Data) has image files named like `DC_DCAM#1_CAM3.tif`.

```python
from obspec_utils.registry import ObjectStoreRegistry
from obstore.store import MemoryStore

reg = ObjectStoreRegistry()
reg.register("s3://bucket", MemoryStore())

store, path = reg.resolve("s3://bucket/path/DC_DCAM#1_CAM3.tif")
print(path) # -> 'path/DC_DCAM' ❌ expected 'path/DC_DCAM#1_CAM3.tif'
```

Downstream this surfaces as a spurious `FileNotFound` for the truncated key. We hit it via `virtual_tiff` → `async-tiff` reading the gallery:

```
AsyncTiffException: FileNotFoundError: Object at location .../Dest210531-152149/DC_DCAM ...
```

### Motivation 2 — presigned URLs / SAS tokens for private buckets

A natural way to give a reader access to a **private** bucket without distributing credentials is to hand VirtualiZarr **presigned URLs** (or Azure SAS URLs), where the signature lives in the query string:

```
https://bucket.s3.us-east-1.amazonaws.com/key?X-Amz-Credential=...&X-Amz-Signature=...
```

`resolve()` currently drops the entire `?...` query, i.e. it **strips the signature**, so every read against a generic HTTP store fails auth (403). Preserving the query is required for this pattern to work at all.

```python
store, path = reg.resolve("https://host/data/file.nc?token=abc123")
print(path) # -> 'data/file.nc' ❌ token lost (expected 'data/file.nc?token=abc123')
```

## Cause

`registry.py`, in `resolve()`:

```python
parsed = urlparse(url)
path = parsed.path # drops fragment ('#...'), query ('?...'), params (';...')
```

## Fix

Reattach params/query/fragment to reconstruct the full key. PR to follow.

Note: for a presigned URL that resolves to an *already-credentialed* object store (rather than a
generic HTTP store), the reattached query would become part of the key — but that is a
contradictory configuration (the store already authenticates), and is out of scope for both
motivations above. Happy to gate on store type if maintainers prefer.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in registry.py at ObjectStoreRegistry.resolve() and reproduce the issue with the object-key and presigned-URL examples from the report. Add regression coverage for keys containing semicolons, question marks, and fragments, and confirm resolution preserves the complete key or URL needed by the relevant store.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.