developmentseed / developmentseed/obspec-utils
resolve() truncates object keys containing '#', '?', or ';' (breaks special-char keys and presigned URLs)
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
`ObjectStoreRegistry.resolve()` derives the object key from `urlparse(url).path`, which silently discards anything `urlparse` treats as URL **params (`;`)**, **query (`?`)**, or **fragment (`#`)**. As a result the resolved key is truncated whenever a URL contains one of those characters.
This bites in two distinct, real situations:
### Motivation 1 — object keys that contain `#` / `?` / `;`
These characters are valid in S3/GCS/Azure object keys. Concrete example: the [Cell Painting Gallery](https://registry.opendata.aws/cellpainting-gallery/) (AWS Open Data) has image files named like `DC_DCAM#1_CAM3.tif`.
```python
from obspec_utils.registry import ObjectStoreRegistry
from obstore.store import MemoryStore
reg = ObjectStoreRegistry()
reg.register("s3://bucket", MemoryStore())
store, path = reg.resolve("s3://bucket/path/DC_DCAM#1_CAM3.tif")
print(path) # -> 'path/DC_DCAM' ❌ expected 'path/DC_DCAM#1_CAM3.tif'
```
Downstream this surfaces as a spurious `FileNotFound` for the truncated key. We hit it via `virtual_tiff` → `async-tiff` reading the gallery:
```
AsyncTiffException: FileNotFoundError: Object at location .../Dest210531-152149/DC_DCAM ...
```
### Motivation 2 — presigned URLs / SAS tokens for private buckets
A natural way to give a reader access to a **private** bucket without distributing credentials is to hand VirtualiZarr **presigned URLs** (or Azure SAS URLs), where the signature lives in the query string:
```
https://bucket.s3.us-east-1.amazonaws.com/key?X-Amz-Credential=...&X-Amz-Signature=...
```
`resolve()` currently drops the entire `?...` query, i.e. it **strips the signature**, so every read against a generic HTTP store fails auth (403). Preserving the query is required for this pattern to work at all.
```python
store, path = reg.resolve("https://host/data/file.nc?token=abc123")
print(path) # -> 'data/file.nc' ❌ token lost (expected 'data/file.nc?token=abc123')
```
## Cause
`registry.py`, in `resolve()`:
```python
parsed = urlparse(url)
path = parsed.path # drops fragment ('#...'), query ('?...'), params (';...')
```
## Fix
Reattach params/query/fragment to reconstruct the full key. PR to follow.
Note: for a presigned URL that resolves to an *already-credentialed* object store (rather than a
generic HTTP store), the reattached query would become part of the key — but that is a
contradictory configuration (the store already authenticates), and is out of scope for both
motivations above. Happy to gate on store type if maintainers prefer.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in registry.py at ObjectStoreRegistry.resolve() and reproduce the issue with the object-key and presigned-URL examples from the report. Add regression coverage for keys containing semicolons, question marks, and fragments, and confirm resolution preserves the complete key or URL needed by the relevant store.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100