apache / apache/arrow

[Format][FlightRPC] Transfer FlightData in pieces

Open
#34,485 2 comments 0 reactions 0 assignees View on GitHub
Component: FlightRPC Component: Format Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

gRPC presents a message-based interface (not a stream-based interface like HTTP). It also limits the size of individual messages by default. For Flight, this means that large record batches will by default cause gRPC clients to error.

Flight clients generally override this setting to avoid this error, but some users may want to use the default gRPC client, and some users may want to keep this setting enabled to protect against misbehaving servers. (And some users may not know about this behavior.) Indeed, the Go Flight client doesn't unlock this setting by default.

For maximal compatibility, Flight servers can reslice record batches to limit their size before sending them on the wire. However, this has a few problems. Estimating the size of a record batch takes some work, and then you don't know the size of the IPC metadata or Protobuf metadata (which gets included in the gRPC message size), so a fudge factor is necessary. Slicing is not necessarily free in all implementations, and it is not free once we get to IPC and may need to rewrite offset buffers. (This should be cheap, but it isn't free.) And a single row is unsliceable, but large string data or highly nested data may put a large amount of data in a single logical row.

In C++ DoPut has a mechanism to work around this on the client by implementing an 'optimistic' check (where the size is checked after IPC serialization, and a distinct error returned so the client can slice and try again), but it is error-prone to implement and isn't currently available to the server or across languages.

An alternative may be to have Flight model the IPC structure in more detail. Currently, a FlightData message is effectively a RecordBatch modeled as an IPC message + a single body buffer. Instead, we could transfer the data as multiple messages with a gRPC/Protobuf message representing one or more buffers. That way the only limitation is that an individual buffer must fit in the message size limit. (So large string data may still be a problem.) (We could even fragment buffers, but then it becomes non-zero-copy on the reader. I don't think there's a way to work around that without implementing gRPC ourselves. We may have to accept that tradeoff if large string data is a problem, though.)

This may also be an opportunity to solve #32276. We should know what the Protobuf metadata looks like and be able to finagle it so that the metadata always takes 8 bytes, or insert our own padding. (However, given gRPC-C++ returns messages as a list of byte slices with no guarantees on alignment as far as I'm aware, just fixing it on the wire format side may not be enough.)

This also touches on prior issues about feature/version negotiation in Flight, since we would need some way to know to use the new format.

### Component(s)

FlightRPC, Format

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the FlightData message, FlightRPC, IPC serialization, and existing feature/version negotiation described in the issue. Determine whether a compatible multi-message wire format can be agreed on across languages, including message-size and alignment constraints. Done requires an accepted design with compatibility and negotiation behavior defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, grpc
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.