`FlightDataEncoder::build` accepts inconsistent schemas
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
**Describe the bug**
Looks like
https://github.com/apache/arrow-rs/blob/8580e858c73eab442deb74d194af31385d78c95c/arrow-flight/src/encode.rs#L149-L153
accepts a stream of `RecordBatches` but then only transmits the first schema:
https://github.com/apache/arrow-rs/blob/8580e858c73eab442deb74d194af31385d78c95c/arrow-flight/src/encode.rs#L260-L264
However the typed interface basically allows the user to pass in `RecordBatches` w/ changing schemas.
**To Reproduce**
\-
**Expected behavior**
One of:
- it is technically impossible to change the schema in-flight (i.e. the parameter to `build(...)` should be a batch w/o a schema)
- it errors when the schema changes
- it re-transmits the schema when it changes.
**Additional context**
Contributor guide
Research direction
Start in arrow-flight/src/encode.rs at the referenced lines 149-153 and 260-264, then trace how FlightDataEncoder::build obtains and transmits schemas. Determine how changing RecordBatch schemas should be handled among the stated alternatives, and verify that the chosen behavior is covered for inconsistent input schemas.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100