apache / apache/sedona-db

python/sedonadb: Add Python extensibility layer for the catalog provider stack

Open
#1,150 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 4h
Merged PRs (30d)
90

Description

While we allow file formats and table providers to be plugged in via Python, we don't have a layer for catalogs. We should add the ability for this!

```python
class MyCatalogProvider(CatalogProvider):

def list_catalogs(self):
...
```

This stack would mirror the CatalogProviderList / CatalogProvider / SchemaProvider stack from DataFusion, although we should arguably make it arbitrarily nestable to support catalogs that allow that.

This is in preparation for supporting providers like Iceberg (REST catalog) that have a catalog-like hierarchy built in. It will also help with things like Overture that benefit from a metadata cache with very specific parameters / schemas that are expensive to compute but are mostly known ahead of time.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing Python-pluggable file-format and table-provider layers, then compare them with DataFusion's CatalogProviderList, CatalogProvider, and SchemaProvider stack. Clarify the proposed CatalogProvider interface, including whether arbitrary nesting is required for Iceberg-style hierarchies; done means a usable Python catalog-provider layer with the intended hierarchy supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.