developmentseed / developmentseed/obstore
Fsspec integration fails for DuckDB
- Dominant language
- Python
- Stars
- 810
- Forks
- 42
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 6
Description
This works with s3fs:
```py
import duckdb
from s3fs import S3FileSystem
fs = S3FileSystem(anon=True)
con = duckdb.connect()
con.register_filesystem(fs)
sql = """\
SELECT
count(*)
FROM read_parquet('s3://overturemaps-us-west-2/release/2025-02-19.0/theme=base/type=land/*', filename=true, hive_partitioning=1)
"""
con.sql(sql)
```
This doesn't work with obstore:
```py
import duckdb
from obstore.fsspec import FsspecStore
fs = FsspecStore("s3", skip_signature=True, region="us-west-2")
con = duckdb.connect()
con.register_filesystem(fs)
sql = """\
SELECT
count(*)
FROM read_parquet('s3://overturemaps-us-west-2/release/2025-02-19.0/theme=base/type=land/*', filename=true, hive_partitioning=1)
"""
con.sql(sql)
```
It errors with
```
---------------------------------------------------------------------------
IOException Traceback (most recent call last)
Cell In[7], [line 1](vscode-notebook-cell:?execution_count=7&line=1)
----> [1](vscode-notebook-cell:?execution_count=7&line=1) con.sql(sql)
IOException: IO Error: No files found that match the pattern "s3://overturemaps-us-west-2/release/2025-02-19.0/theme=base/type=land/*"
```
cc @machichima you may be curious about this because it signifies an incompatibility with fsspec that you might hit as well
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.