apache / apache/arrow-flight-sql-postgresql

Improve `SELECT text` performance

Open
#177 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
116
Forks
10
PR merge metrics
No merged PRs in 30d

Description

It's slower than the PostgreSQL protocol.

The followings may be related but we need to look into it:

* Building `arrow::RecordBatch` is slow?
* We need to build `arrow::RecordBatch`es to use `arrow::ipc::RecordBatchWriter()`. We need to copy PostgreSQL data for it. (Not zero-copy.)
* Should we add an API that writes Apache Arrow streaming format data without building `arrow::RecordBatch` to Apache Arrow C++?
* Calling `SPI_getbinval()` is slow?
* It calls `nocachegetattr()` https://github.com/postgres/postgres/blob/3edc6580c0e27fb8f13322efd255a88d20dda6c2/src/backend/access/common/heaptuple.c#L496-L712 and it's not a short function. Can we shortcut some operations?

Contributor guide

Open the contributing guide

Research direction

Start by profiling the SELECT text path and comparing it with the PostgreSQL protocol, focusing on Apache Arrow RecordBatch construction, RecordBatchWriter, and PostgreSQL's SPI_getbinval(). Determine whether copying into RecordBatch or SPI_getbinval()/nocachegetattr() dominates, then measure the completed change against the PostgreSQL protocol baseline.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, postgresql
Domain
backend, databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.