developmentseed / developmentseed/obstore

Fsspec integration fails for DuckDB

Open
#328 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
810
Forks
42
Avg merge
1d 15h
Merged PRs (30d)
6

Description

This works with s3fs:

```py
import duckdb
from s3fs import S3FileSystem

fs = S3FileSystem(anon=True)
con = duckdb.connect()
con.register_filesystem(fs)

sql = """\
SELECT
count(*)
FROM read_parquet('s3://overturemaps-us-west-2/release/2025-02-19.0/theme=base/type=land/*', filename=true, hive_partitioning=1)
"""

con.sql(sql)
```

This doesn't work with obstore:

```py
import duckdb

from obstore.fsspec import FsspecStore

fs = FsspecStore("s3", skip_signature=True, region="us-west-2")
con = duckdb.connect()
con.register_filesystem(fs)

sql = """\
SELECT
count(*)
FROM read_parquet('s3://overturemaps-us-west-2/release/2025-02-19.0/theme=base/type=land/*', filename=true, hive_partitioning=1)
"""

con.sql(sql)
```
It errors with
```
---------------------------------------------------------------------------
IOException Traceback (most recent call last)
Cell In[7], [line 1](vscode-notebook-cell:?execution_count=7&line=1)
----> [1](vscode-notebook-cell:?execution_count=7&line=1) con.sql(sql)

IOException: IO Error: No files found that match the pattern "s3://overturemaps-us-west-2/release/2025-02-19.0/theme=base/type=land/*"
```

cc @machichima you may be curious about this because it signifies an incompatibility with fsspec that you might hit as well

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.