developmentseed / developmentseed/obstore

`open_reader` double-applies a store `prefix=`

Open
#744 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
810
Forks
42
Avg merge
1d 15h
Merged PRs (30d)
6

Description

On a store created with a `prefix=` (e.g. `S3Store(prefix="P")`), `obstore.open_reader(store, "key")` resolves the object at `P/P/key` instead of `P/key`, so every buffered read of a prefixed store fails with a 404. `get`, `put`, and `list` on the same store are correct (single prefix). Regressed in 0.9.0; still present in 0.11.0.

## Environment

- obstore 0.11.0 also reproduced on 0.9.0.
- Last good: 0.8.2

## Reproduction

Standalone `uv run repro.py` (deps declared inline; moto provides a real S3 backend). The object is written once at `P/key`; swap the `obstore` pin to `0.8.2` to see the correct behavior.

```python
# /// script
# requires-python = ">=3.11"
# dependencies = ["obstore==0.11.0", "boto3==1.43.22", "moto[server]==5.2.1"]
# ///
import boto3, obstore
from moto.server import ThreadedMotoServer
from obstore.store import S3Store

server = ThreadedMotoServer(port=0); server.start()
host, port = server.get_host_and_port()
endpoint = f"http://{host}:{port}"

s3 = boto3.client("s3", endpoint_url=endpoint, region_name="us-east-1",
aws_access_key_id="x", aws_secret_access_key="x")
s3.create_bucket(Bucket="bucket")
s3.put_object(Bucket="bucket", Key="P/key", Body=b"hello") # object lives at P/key

store = S3Store(bucket="bucket", prefix="P", endpoint=endpoint, region="us-east-1",
access_key_id="x", secret_access_key="x",
client_options={"allow_http": True}, virtual_hosted_style_request=False)

print("list :", [o["path"] for b in obstore.list(store) for o in b]) # ['key']
print("head :", obstore.head(store, "key")["path"]) # 'P/key' <- should be 'key'
print("get :", bytes(obstore.get(store, "key").bytes())) # b'hello'
try:
print("read :", obstore.open_reader(store, "key").read()) # -> GET P/P/key -> 404
except Exception as e:
print("read :", str(e).splitlines()[0])
server.stop()
```

## Output

**obstore 0.11.0 (bug)** — moto also logs `HEAD /bucket/P/key` then `GET /bucket/P/P/key`:

```
list : ['key'] # prefix stripped (correct)
head : P/key # <- prefix NOT stripped (root cause)
get : b'hello' # correct
read : Object at location P/P/key not found: ... GET .../bucket/P/P/key ... 404
```

**obstore 0.8.2 (correct)** — GET goes to `/bucket/P/key`:

```
list : ['key']
head : key
get : b'hello'
read : Bytes(b'hello')
```

## Expected

`open_reader(store, "key")` on a store with `prefix="P"` should read `P/key` (as `get`/`put`/`list` do), and `head(...)["path"]` should return `"key"` (prefix-relative), consistent with `list()`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the standalone repro.py and compare the path handling used by open_reader, head, get, and list for an S3Store with prefix="P". Trace how open_reader consumes the head result, then verify that reads use P/key and head(...)["path"] returns key, matching the expected behavior and the 0.8.2 output.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.