Decouple the `FileSource` trait from the concrete `FileScanConfig` `DataSource`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I'd like to build a custom `DataSource` that reuses an existing `FileSource` (e.g. `ParquetSource`) instead of reimplementing all its internals. Today this is not possible without dragging along a `FileScanConfig` that my custom `DataSource` doesn't otherwise need.
The architecture diagram in `datafusion/datasource/src/source.rs` shows a clean top-down layering:
```text
DataSourceExec ──► DataSource (trait) ──► FileScanConfig ──► FileSource (trait) ──► ParquetSource / ...
```
Roughly speaking, the`DataSource` handles execution and partitioning and the `FileSource` handles file-type specifics. But `FileSource` is currently coupled to `FileScanConfig` (the concrete `DataSource`). Its open-time methods take a `&FileScanConfig` and use it as a "bag of args" (e.g. `ParquetSource` pulls `expr_adapter_factory`, `limit`, `preserve_order`, ...).
```rust
fn create_file_opener(&self, object_store: Arc, base_config: &FileScanConfig, partition: usize) -> ...;
fn create_morselizer(&self, object_store: Arc, base_config: &FileScanConfig, partition: usize) -> ...;
```
Decoupling the two would let a custom `DataSource` reuse any `FileSource` directly.
### Describe the solution you'd like
Pass those inputs explicitly via a small, freely-constructible args struct, replacing retrieval direct from the `&FileScanConfig`:
```rust
#[derive(Debug, Clone)]
#[non_exhaustive]
pub struct FileSourceArgs {
pub object_store: Arc,
pub batch_size: usize,
pub limit: Option,
pub preserve_order: bool,
pub file_compression_type: FileCompressionType,
pub expr_adapter_factory: Option>,
}
fn create_file_opener(&self, args: &FileSourceArgs, partition: usize) -> ...;
fn create_morselizer(&self, args: &FileSourceArgs, partition: usize) -> ...;
// with_batch_size removed — batch size now flows through FileSourceArgs
```
`FileScanConfig` builds `FileSourceArgs` inline (behavior unchanged). However, now a custom `DataSource` can build the same struct and reuse any `FileSource` too.
This is a **breaking change** to the public `FileSource` trait, but the migration path is simple.
I have a draft PR ready for this issue.
### Describe alternatives you've considered
1. Do nothing. I can construct a dummy `FileScanConfig` within my custom `DataSource` - simply to satisfy the interface - but the construction is quite messy.
2. Extend the `DataSource` trait with all getter fns that a `FileSource` may - might(?) avoid a breaking change, but adds coupling that may not make sense for other custom DataSources.
3. Fork ParquetSource, add a custom `fn create_morselizer()` that doesn't depend on `FileSourceArgs` - I'd prefer not to diverge from upstream
### Additional context
The goal of my custom DataSource is to increase I/O parallelism on an I/O bound workload with plenty of CPU to spare. To achieve this, I am diverging from the `FileStream` currently tightly coupled in a `FileScanConfig`.
P.S. this is my first datafusion issue. Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.