apache / apache/arrow-rs

RecordBatchWriter trait can't be used in dynamic context due to close(self)

Open
#6,326 7 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

I'm trying to use [`RecordBatchWriter`](https://docs.rs/arrow/52.2.0/arrow/array/trait.RecordBatchWriter.html) trait to introduce dynamism in the type of output - e.g. depending on the command line options, I'd like to write parquet, csv, json or ipc. I know that corresponding writers all implement this trait, thanks to #4206 and #4228 by @alexandreyc.

Specifically, I was planning to use `Box` and pass it to the writing code, something like this:
```rust
let writer: Box = match args.format {
"parquet" => Box::new(parquet::arrow::ArrowWriter::try_new(file, schema, None)),
"csv" => Box::new(arrow::csv::Writer::new(file)),
...
}

writer.write(my_batch);
writer.close(); // this is required to write the formats correctly
```

Unfortunately, due to `close` method in that trait consuming `self`, we can't really use this approach, as now close() needs to know the exact type of the writer. Calling close() on the trait object gives compilation error "the size of `dyn RecordBatchWriter` cannot be statically determined". See [example in Rust playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=2f5a6a45fb2c4321dbb4806009bafaa3).

**Describe the solution you'd like**

Add a `finish(&mut self)` method to the trait. This is backwards-compatible, as well as an established pattern e.g. see [parquet writers](https://docs.rs/parquet/52.2.0/parquet/arrow/arrow_writer/struct.ArrowWriter.html#method.finish).

This will not make the trait object-safe, but will unblock usage as `Box`.

**Describe alternatives you've considered**

Manually downcasting to specific types and then calling close() on them, but that doesn't seem to work, as RecordBatchWriter needs to explicitly have `as_any` method or similar.

Contributor guide

Open the contributing guide

Research direction

Start with the RecordBatchWriter trait in the Arrow Rust API and review the parquet ArrowWriter and CSV Writer implementations referenced in the issue. Check the Rust playground example to reproduce the dyn RecordBatchWriter failure. Done means the requested mutable finishing entry point is available for dynamic writers and the affected implementations continue to finish output correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.