read_parquet rejects URLs without a .parquet extension and offers no way to bypass the check
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
## Problem
`read_parquet` refuses any path that does not end in `.parquet`, and there is no way to relax or bypass that check. This breaks reading GeoParquet from APIs that serve it from a URL without a file extension, for example OGC API - Features style endpoints:
```python
import sedona.db
sd = sedona.db.connect()
df = sd.read_parquet("https://example.com/collections/collection_id/items?f=parquet")
```
```
sedonadb._lib.SedonaError: File path 'https://example.com/collections/collection_id/items?f=parquet' does not match the expected extension '.parquet'
```
Reported on Discord: https://discord.com/channels/1034906732264181760/1545393838678478888
## Where the check lives
`geoparquet_listing_table` in `rust/sedona-geoparquet/src/provider.rs` strips the query string and then requires the remaining path to end in `listing_options.file_extension`, which is always the `ParquetReadOptions` default of `.parquet`. `GeoParquetReadOptions` never exposes `file_extension`, and the `options` dict from Python only feeds DataFusion `TableOptions` (`aws.*`, `azure.*`, `format.*`), so neither `{"file_extension": ""}` nor `{"format.file_extension": ""}` has any effect.
`sd.read()` has the same limitation on the Python side because it guesses the format from the extension and raises `Can't guess format from paths where no item has an extension`.
## Workaround that works today
The SQL path does not apply the check, and the resulting table still goes through the GeoParquet format (geometry column typed with CRS, `ST_*` functions work):
```python
sd.sql("""
CREATE EXTERNAL TABLE items
STORED AS PARQUET
LOCATION 'https://example.com/collections/collection_id/items?f=parquet'
""")
sd.sql("SELECT * FROM items").show()
```
Verified against a GeoParquet file served over HTTP from a URL with no extension and a `?f=parquet` query string (the server must support Range requests, as with any remote Parquet read).
## Proposal
Expose a way to skip or override the extension check in `read_parquet`:
- Rust: add `file_extension` (or a `check_extension: bool`) to `GeoParquetReadOptions` and thread it into `ParquetReadOptions` / the check in `geoparquet_listing_table`. `sedona-datasource` already has a `check_extension` flag on `RecordBatchReaderTableOptions`, so GeoParquet is currently the odd one out.
- Python: accept it in `SedonaContext.read_parquet(...)`, either as a keyword argument or as a recognized key in `options`.
- Optionally let `sd.read()` accept an explicit `format=` so it does not have to guess from the path.
Contributor guide
Research direction
Start in rust/sedona-geoparquet/src/provider.rs at geoparquet_listing_table and trace GeoParquetReadOptions through the Python SedonaContext.read_parquet entry point. Compare the existing check_extension flag in sedona-datasource, then verify that the chosen interface lets read_parquet load a URL without a .parquet suffix while retaining GeoParquet geometry and CRS behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100