python/sedonadb: Add Python extensibility layer for the catalog provider stack
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
While we allow file formats and table providers to be plugged in via Python, we don't have a layer for catalogs. We should add the ability for this!
```python
class MyCatalogProvider(CatalogProvider):
def list_catalogs(self):
...
```
This stack would mirror the CatalogProviderList / CatalogProvider / SchemaProvider stack from DataFusion, although we should arguably make it arbitrarily nestable to support catalogs that allow that.
This is in preparation for supporting providers like Iceberg (REST catalog) that have a catalog-like hierarchy built in. It will also help with things like Overture that benefit from a metadata cache with very specific parameters / schemas that are expensive to compute but are mostly known ahead of time.
Contributor guide
Research direction
Start by tracing the existing Python-pluggable file-format and table-provider layers, then compare them with DataFusion's CatalogProviderList, CatalogProvider, and SchemaProvider stack. Clarify the proposed CatalogProvider interface, including whether arbitrary nesting is required for Iceberg-style hierarchies; done means a usable Python catalog-provider layer with the intended hierarchy supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100