influxdata / influxdata/influxdb
Stream query response for Parquet format
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
### Problem statement
When a query is made via the HTTP API (`/api/v3/query_sql`, `/api/v3/query_influxql`) with `parquet` as the output format, the record batches in the response are buffered into memory before being serialized to parquet.
This could lead to OOMs for large query responses.
### Additional context
The [parquet writer](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowWriter.html) is not compatible with writing to a streamed `Body` using the current version of hyper that we are using.
We either need to upgrade to hyper 1.x, to see if its trait-based approach to [`Body`](https://docs.rs/hyper/latest/hyper/body/trait.Body.html) can be used alongside the existing arrow writers, or there may be some upstream work in `arrow-rs` required to make this possible.
#### Related:
* https://github.com/influxdata/influxdb/issues/24655
* https://github.com/influxdata/influxdb/pull/25927
### Proposed solution
We should first start by upgrading `hyper` to 1.x. This may need to be done upstream in IOx before it can be brought in here, since we are leveraging code from IOx that uses hyper 0.14.x.
Then we can see if implementing a streamed writer for parquet with the new `Body` trait is possible, or if we need to open some issues upstream in `arrow-rs`.
Contributor guide
Research direction
Start with the HTTP API endpoints /api/v3/query_sql and /api/v3/query_influxql, then trace how parquet responses use ArrowWriter and how the current hyper 0.14 Body is produced. Investigate the proposed hyper 1.x upgrade and related IOx and arrow-rs constraints; done means parquet record batches are streamed without buffering the full response in memory, including for large queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100