RecordBatchWriter trait can't be used in dynamic context due to close(self)
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm trying to use [`RecordBatchWriter`](https://docs.rs/arrow/52.2.0/arrow/array/trait.RecordBatchWriter.html) trait to introduce dynamism in the type of output - e.g. depending on the command line options, I'd like to write parquet, csv, json or ipc. I know that corresponding writers all implement this trait, thanks to #4206 and #4228 by @alexandreyc.
Specifically, I was planning to use `Box` and pass it to the writing code, something like this:
```rust
let writer: Box = match args.format {
"parquet" => Box::new(parquet::arrow::ArrowWriter::try_new(file, schema, None)),
"csv" => Box::new(arrow::csv::Writer::new(file)),
...
}
writer.write(my_batch);
writer.close(); // this is required to write the formats correctly
```
Unfortunately, due to `close` method in that trait consuming `self`, we can't really use this approach, as now close() needs to know the exact type of the writer. Calling close() on the trait object gives compilation error "the size of `dyn RecordBatchWriter` cannot be statically determined". See [example in Rust playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=2f5a6a45fb2c4321dbb4806009bafaa3).
**Describe the solution you'd like**
Add a `finish(&mut self)` method to the trait. This is backwards-compatible, as well as an established pattern e.g. see [parquet writers](https://docs.rs/parquet/52.2.0/parquet/arrow/arrow_writer/struct.ArrowWriter.html#method.finish).
This will not make the trait object-safe, but will unblock usage as `Box`.
**Describe alternatives you've considered**
Manually downcasting to specific types and then calling close() on them, but that doesn't seem to work, as RecordBatchWriter needs to explicitly have `as_any` method or similar.
Contributor guide
Research direction
Start with the RecordBatchWriter trait in the Arrow Rust API and review the parquet ArrowWriter and CSV Writer implementations referenced in the issue. Check the Rust playground example to reproduce the dyn RecordBatchWriter failure. Done means the requested mutable finishing entry point is available for dynamic writers and the affected implementations continue to finish output correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100