apache / apache/sedona-db

read_parquet rejects URLs without a .parquet extension and offers no way to bypass the check

Open
#1,245 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 4h
Merged PRs (30d)
90

Description

## Problem

`read_parquet` refuses any path that does not end in `.parquet`, and there is no way to relax or bypass that check. This breaks reading GeoParquet from APIs that serve it from a URL without a file extension, for example OGC API - Features style endpoints:

```python
import sedona.db

sd = sedona.db.connect()
df = sd.read_parquet("https://example.com/collections/collection_id/items?f=parquet")
```

```
sedonadb._lib.SedonaError: File path 'https://example.com/collections/collection_id/items?f=parquet' does not match the expected extension '.parquet'
```

Reported on Discord: https://discord.com/channels/1034906732264181760/1545393838678478888

## Where the check lives

`geoparquet_listing_table` in `rust/sedona-geoparquet/src/provider.rs` strips the query string and then requires the remaining path to end in `listing_options.file_extension`, which is always the `ParquetReadOptions` default of `.parquet`. `GeoParquetReadOptions` never exposes `file_extension`, and the `options` dict from Python only feeds DataFusion `TableOptions` (`aws.*`, `azure.*`, `format.*`), so neither `{"file_extension": ""}` nor `{"format.file_extension": ""}` has any effect.

`sd.read()` has the same limitation on the Python side because it guesses the format from the extension and raises `Can't guess format from paths where no item has an extension`.

## Workaround that works today

The SQL path does not apply the check, and the resulting table still goes through the GeoParquet format (geometry column typed with CRS, `ST_*` functions work):

```python
sd.sql("""
CREATE EXTERNAL TABLE items
STORED AS PARQUET
LOCATION 'https://example.com/collections/collection_id/items?f=parquet'
""")
sd.sql("SELECT * FROM items").show()
```

Verified against a GeoParquet file served over HTTP from a URL with no extension and a `?f=parquet` query string (the server must support Range requests, as with any remote Parquet read).

## Proposal

Expose a way to skip or override the extension check in `read_parquet`:

- Rust: add `file_extension` (or a `check_extension: bool`) to `GeoParquetReadOptions` and thread it into `ParquetReadOptions` / the check in `geoparquet_listing_table`. `sedona-datasource` already has a `check_extension` flag on `RecordBatchReaderTableOptions`, so GeoParquet is currently the odd one out.
- Python: accept it in `SedonaContext.read_parquet(...)`, either as a keyword argument or as a recognized key in `options`.
- Optionally let `sd.read()` accept an explicit `format=` so it does not have to guess from the path.

Contributor guide

Open the contributing guide

Research direction

Start in rust/sedona-geoparquet/src/provider.rs at geoparquet_listing_table and trace GeoParquetReadOptions through the Python SedonaContext.read_parquet entry point. Compare the existing check_extension flag in sedona-datasource, then verify that the chosen interface lets read_parquet load a URL without a .parquet suffix while retaining GeoParquet geometry and CRS behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.