apache / apache/sedona-db

Ensure FileFormat can be implemented in extension packages

Open
#1,205 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 5h
Merged PRs (30d)
91

Description

Currently we have a fairly good way to implement a TableProvider; however we don't have a good way to deal with extensions that want to deal with files and benefit from automatic listing and partitioning like is done for Parquet.

This is a prerequisite for publishing an extension package for las/laz read and other future sources. It would benefit from being able to handle vendored credentials (e.g., so that extensions don't have to also repackage aws libraries).

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing TableProvider and Parquet implementations and tracing how file listing, partitioning, and credentials are handled. Define an extension-facing FileFormat API that supports las/laz and future file sources, including vendored credentials, then verify that an extension package can use it without repackaging AWS libraries.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.