developmentseed / developmentseed/obstore
`open_reader` double-applies a store `prefix=`
- Dominant language
- Python
- Stars
- 810
- Forks
- 42
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 6
Description
On a store created with a `prefix=` (e.g. `S3Store(prefix="P")`), `obstore.open_reader(store, "key")` resolves the object at `P/P/key` instead of `P/key`, so every buffered read of a prefixed store fails with a 404. `get`, `put`, and `list` on the same store are correct (single prefix). Regressed in 0.9.0; still present in 0.11.0.
## Environment
- obstore 0.11.0 also reproduced on 0.9.0.
- Last good: 0.8.2
## Reproduction
Standalone `uv run repro.py` (deps declared inline; moto provides a real S3 backend). The object is written once at `P/key`; swap the `obstore` pin to `0.8.2` to see the correct behavior.
```python
# /// script
# requires-python = ">=3.11"
# dependencies = ["obstore==0.11.0", "boto3==1.43.22", "moto[server]==5.2.1"]
# ///
import boto3, obstore
from moto.server import ThreadedMotoServer
from obstore.store import S3Store
server = ThreadedMotoServer(port=0); server.start()
host, port = server.get_host_and_port()
endpoint = f"http://{host}:{port}"
s3 = boto3.client("s3", endpoint_url=endpoint, region_name="us-east-1",
aws_access_key_id="x", aws_secret_access_key="x")
s3.create_bucket(Bucket="bucket")
s3.put_object(Bucket="bucket", Key="P/key", Body=b"hello") # object lives at P/key
store = S3Store(bucket="bucket", prefix="P", endpoint=endpoint, region="us-east-1",
access_key_id="x", secret_access_key="x",
client_options={"allow_http": True}, virtual_hosted_style_request=False)
print("list :", [o["path"] for b in obstore.list(store) for o in b]) # ['key']
print("head :", obstore.head(store, "key")["path"]) # 'P/key' <- should be 'key'
print("get :", bytes(obstore.get(store, "key").bytes())) # b'hello'
try:
print("read :", obstore.open_reader(store, "key").read()) # -> GET P/P/key -> 404
except Exception as e:
print("read :", str(e).splitlines()[0])
server.stop()
```
## Output
**obstore 0.11.0 (bug)** — moto also logs `HEAD /bucket/P/key` then `GET /bucket/P/P/key`:
```
list : ['key'] # prefix stripped (correct)
head : P/key # <- prefix NOT stripped (root cause)
get : b'hello' # correct
read : Object at location P/P/key not found: ... GET .../bucket/P/P/key ... 404
```
**obstore 0.8.2 (correct)** — GET goes to `/bucket/P/key`:
```
list : ['key']
head : key
get : b'hello'
read : Bytes(b'hello')
```
## Expected
`open_reader(store, "key")` on a store with `prefix="P"` should read `P/key` (as `get`/`put`/`list` do), and `head(...)["path"]` should return `"key"` (prefix-relative), consistent with `list()`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the standalone repro.py and compare the path handling used by open_reader, head, get, and list for an S3Store with prefix="P". Trace how open_reader consumes the head result, then verify that reads use P/key and head(...)["path"] returns key, matching the expected behavior and the 0.8.2 output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100