`FileSource` and `DataSource` traits require deep copies
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
While working on various upgrade PRs and preparing for the DataFusion 46 release, I have noticed something I would like to change before we release
The `FileSource` and `DataSource` traits were introduced in the datasource refactor
- https://github.com/apache/datafusion/pull/14224
They have APIs to update the underlying source in a few ways, but the APIs require cloning. For example, `FileSource` looks like this:
```rust
/// Common behaviors that every file format needs to implement.
///
/// See initialization examples on `ParquetSource`, `CsvSource`
pub trait FileSource: Send + Sync {
...
/// Initialize new type with batch size configuration
fn with_batch_size(&self, batch_size: usize) -> Arc;
...
}
```
The only way to implement `with_batch_size` is to (deep) clone the object
```rust
fn with_batch_size(&self, batch_size: usize) -> Arc {
let mut conf = self.clone();
conf.batch_size = Some(batch_size);
Arc::new(conf)
}
```
https://github.com/apache/datafusion/blob/1ae06a497e7c6b117c211c52b33445c2063b9921/datafusion/core/src/datasource/physical_plan/csv.rs#L584-L588
### Describe the solution you'd like
I would like to avoid having to deep clone the object
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the FileSource and DataSource trait definitions introduced by datasource refactor PR #14224, then inspect the CSV implementation at datafusion/core/src/datasource/physical_plan/csv.rs around lines 584-588. Focus on how with_batch_size and related update APIs currently clone sources; done means an agreed API change is implemented across affected sources and the existing tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100