apache / apache/datafusion

request: Connect file groups in datasource to their object store

Open
#16,991 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Currently, a filegroup will contain partitioned files, which when created from a path, will have the scheme and authority stripped from them in the parsing.
The datasourceexec itself only supports a single objectstoreurl/objectstore
therefore, if I create a filegroup from the following partitioned files
"s3://my-bucket/file" and "file:///my-local-file", I would get a file not found error from the object store, since the datasource itself can only be created with a single object store, either for the s3 or file.
For my use case, my partitioner found that the best partitioning scheme was to load files from those two locations in one partition, and the current exec doesn't let me do that.

### Describe the solution you'd like

For the datasource exec to maintain a map of ObjectStoreUrl -> ObjectStore, or just a set of registered ObjectStoreUrls which can then be used to fetch the object store from the task context.
And for the PartitionedFile itself to maintain an ObjectStoreUrl, as well as the file Path, so the right object store be selected for the file opener.

Additionally, when the datasource exec is created, it should verify that all the partitionedfiles's objectstore is available for the operator.

### Describe alternatives you've considered

Currently I need to either manually fix the partitioning, which is not great for my flow, or manually create a datasourceexec for each file/store and execute it, then have an operator to manually inject each record batch into the rest of the streams, but this ruins the ease-of-use for the datasourceexec's partitioning mechanism which would automatically generate the stream for each partition.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the datasource exec and PartitionedFile implementation described in the issue, then trace how object stores are selected from the task context. Confirm how mixed object-store URLs are currently handled and how datasource creation validates availability; done means partitioned files from multiple stores can be opened by the correct store and unavailable stores are rejected.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.