influxdata / influxdata/influxdb

Stream query response for Parquet format

Open
#25,955 3 comments 0 reactions 0 assignees View on GitHub
v3
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

### Problem statement

When a query is made via the HTTP API (`/api/v3/query_sql`, `/api/v3/query_influxql`) with `parquet` as the output format, the record batches in the response are buffered into memory before being serialized to parquet.

This could lead to OOMs for large query responses.

### Additional context

The [parquet writer](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowWriter.html) is not compatible with writing to a streamed `Body` using the current version of hyper that we are using.

We either need to upgrade to hyper 1.x, to see if its trait-based approach to [`Body`](https://docs.rs/hyper/latest/hyper/body/trait.Body.html) can be used alongside the existing arrow writers, or there may be some upstream work in `arrow-rs` required to make this possible.

#### Related:
* https://github.com/influxdata/influxdb/issues/24655
* https://github.com/influxdata/influxdb/pull/25927

### Proposed solution

We should first start by upgrading `hyper` to 1.x. This may need to be done upstream in IOx before it can be brought in here, since we are leveraging code from IOx that uses hyper 0.14.x.

Then we can see if implementing a streamed writer for parquet with the new `Body` trait is possible, or if we need to open some issues upstream in `arrow-rs`.

Contributor guide

Open the contributing guide

Research direction

Start with the HTTP API endpoints /api/v3/query_sql and /api/v3/query_influxql, then trace how parquet responses use ArrowWriter and how the current hyper 0.14 Body is produced. Investigate the proposed hyper 1.x upgrade and related IOx and arrow-rs constraints; done means parquet record batches are streamed without buffering the full response in memory, including for large queries.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.