apache / apache/arrow-rs

`FlightDataEncoder::build` accepts inconsistent schemas

Open
#4,234 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

**Describe the bug**
Looks like

https://github.com/apache/arrow-rs/blob/8580e858c73eab442deb74d194af31385d78c95c/arrow-flight/src/encode.rs#L149-L153

accepts a stream of `RecordBatches` but then only transmits the first schema:

https://github.com/apache/arrow-rs/blob/8580e858c73eab442deb74d194af31385d78c95c/arrow-flight/src/encode.rs#L260-L264

However the typed interface basically allows the user to pass in `RecordBatches` w/ changing schemas.

**To Reproduce**
\-

**Expected behavior**
One of:

- it is technically impossible to change the schema in-flight (i.e. the parameter to `build(...)` should be a batch w/o a schema)
- it errors when the schema changes
- it re-transmits the schema when it changes.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start in arrow-flight/src/encode.rs at the referenced lines 149-153 and 260-264, then trace how FlightDataEncoder::build obtains and transmits schemas. Determine how changing RecordBatch schemas should be handled among the stated alternatives, and verify that the chosen behavior is covered for inconsistent input schemas.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.