request: Connect file groups in datasource to their object store
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Currently, a filegroup will contain partitioned files, which when created from a path, will have the scheme and authority stripped from them in the parsing.
The datasourceexec itself only supports a single objectstoreurl/objectstore
therefore, if I create a filegroup from the following partitioned files
"s3://my-bucket/file" and "file:///my-local-file", I would get a file not found error from the object store, since the datasource itself can only be created with a single object store, either for the s3 or file.
For my use case, my partitioner found that the best partitioning scheme was to load files from those two locations in one partition, and the current exec doesn't let me do that.
### Describe the solution you'd like
For the datasource exec to maintain a map of ObjectStoreUrl -> ObjectStore, or just a set of registered ObjectStoreUrls which can then be used to fetch the object store from the task context.
And for the PartitionedFile itself to maintain an ObjectStoreUrl, as well as the file Path, so the right object store be selected for the file opener.
Additionally, when the datasource exec is created, it should verify that all the partitionedfiles's objectstore is available for the operator.
### Describe alternatives you've considered
Currently I need to either manually fix the partitioning, which is not great for my flow, or manually create a datasourceexec for each file/store and execute it, then have an operator to manually inject each record batch into the rest of the streams, but this ruins the ease-of-use for the datasourceexec's partitioning mechanism which would automatically generate the stream for each partition.
### Additional context
_No response_
Contributor guide
Research direction
Start by locating the datasource exec and PartitionedFile implementation described in the issue, then trace how object stores are selected from the task context. Confirm how mixed object-store URLs are currently handled and how datasource creation validates availability; done means partitioned files from multiple stores can be opened by the correct store and unavailable stores are rejected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100