apache / apache/datafusion

`FileSource` and `DataSource` traits require deep copies

Open
#14,939 4 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

While working on various upgrade PRs and preparing for the DataFusion 46 release, I have noticed something I would like to change before we release

The `FileSource` and `DataSource` traits were introduced in the datasource refactor
- https://github.com/apache/datafusion/pull/14224

They have APIs to update the underlying source in a few ways, but the APIs require cloning. For example, `FileSource` looks like this:

```rust
/// Common behaviors that every file format needs to implement.
///
/// See initialization examples on `ParquetSource`, `CsvSource`
pub trait FileSource: Send + Sync {
...
/// Initialize new type with batch size configuration
fn with_batch_size(&self, batch_size: usize) -> Arc;
...
}
```

The only way to implement `with_batch_size` is to (deep) clone the object

```rust
fn with_batch_size(&self, batch_size: usize) -> Arc {
let mut conf = self.clone();
conf.batch_size = Some(batch_size);
Arc::new(conf)
}
```

https://github.com/apache/datafusion/blob/1ae06a497e7c6b117c211c52b33445c2063b9921/datafusion/core/src/datasource/physical_plan/csv.rs#L584-L588

### Describe the solution you'd like

I would like to avoid having to deep clone the object

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the FileSource and DataSource trait definitions introduced by datasource refactor PR #14224, then inspect the CSV implementation at datafusion/core/src/datasource/physical_plan/csv.rs around lines 584-588. Focus on how with_batch_size and related update APIs currently clone sources; done means an agreed API change is implemented across affected sources and the existing tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.