apache / apache/datafusion

Decouple the `FileSource` trait from the concrete `FileScanConfig` `DataSource`

Open
#23,463 4 comments 0 reactions 1 assignee Claimed by @mitchsw View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

I'd like to build a custom `DataSource` that reuses an existing `FileSource` (e.g. `ParquetSource`) instead of reimplementing all its internals. Today this is not possible without dragging along a `FileScanConfig` that my custom `DataSource` doesn't otherwise need.

The architecture diagram in `datafusion/datasource/src/source.rs` shows a clean top-down layering:

```text
DataSourceExec ──► DataSource (trait) ──► FileScanConfig ──► FileSource (trait) ──► ParquetSource / ...
```

Roughly speaking, the`DataSource` handles execution and partitioning and the `FileSource` handles file-type specifics. But `FileSource` is currently coupled to `FileScanConfig` (the concrete `DataSource`). Its open-time methods take a `&FileScanConfig` and use it as a "bag of args" (e.g. `ParquetSource` pulls `expr_adapter_factory`, `limit`, `preserve_order`, ...).

```rust
fn create_file_opener(&self, object_store: Arc, base_config: &FileScanConfig, partition: usize) -> ...;
fn create_morselizer(&self, object_store: Arc, base_config: &FileScanConfig, partition: usize) -> ...;
```

Decoupling the two would let a custom `DataSource` reuse any `FileSource` directly.

### Describe the solution you'd like

Pass those inputs explicitly via a small, freely-constructible args struct, replacing retrieval direct from the `&FileScanConfig`:

```rust
#[derive(Debug, Clone)]
#[non_exhaustive]
pub struct FileSourceArgs {
pub object_store: Arc,
pub batch_size: usize,
pub limit: Option,
pub preserve_order: bool,
pub file_compression_type: FileCompressionType,
pub expr_adapter_factory: Option>,
}

fn create_file_opener(&self, args: &FileSourceArgs, partition: usize) -> ...;
fn create_morselizer(&self, args: &FileSourceArgs, partition: usize) -> ...;
// with_batch_size removed — batch size now flows through FileSourceArgs
```

`FileScanConfig` builds `FileSourceArgs` inline (behavior unchanged). However, now a custom `DataSource` can build the same struct and reuse any `FileSource` too.

This is a **breaking change** to the public `FileSource` trait, but the migration path is simple.

I have a draft PR ready for this issue.

### Describe alternatives you've considered

1. Do nothing. I can construct a dummy `FileScanConfig` within my custom `DataSource` - simply to satisfy the interface - but the construction is quite messy.
2. Extend the `DataSource` trait with all getter fns that a `FileSource` may - might(?) avoid a breaking change, but adds coupling that may not make sense for other custom DataSources.
3. Fork ParquetSource, add a custom `fn create_morselizer()` that doesn't depend on `FileSourceArgs` - I'd prefer not to diverge from upstream

### Additional context

The goal of my custom DataSource is to increase I/O parallelism on an I/O bound workload with plenty of CPU to spare. To achieve this, I am diverging from the `FileStream` currently tightly coupled in a `FileScanConfig`.

P.S. this is my first datafusion issue. Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.