apache / apache/arrow-flight-sql-postgresql
Improve `SELECT text` performance
- Dominant language
- C++
- Stars
- 116
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
It's slower than the PostgreSQL protocol.
The followings may be related but we need to look into it:
* Building `arrow::RecordBatch` is slow?
* We need to build `arrow::RecordBatch`es to use `arrow::ipc::RecordBatchWriter()`. We need to copy PostgreSQL data for it. (Not zero-copy.)
* Should we add an API that writes Apache Arrow streaming format data without building `arrow::RecordBatch` to Apache Arrow C++?
* Calling `SPI_getbinval()` is slow?
* It calls `nocachegetattr()` https://github.com/postgres/postgres/blob/3edc6580c0e27fb8f13322efd255a88d20dda6c2/src/backend/access/common/heaptuple.c#L496-L712 and it's not a short function. Can we shortcut some operations?
Contributor guide
Research direction
Start by profiling the SELECT text path and comparing it with the PostgreSQL protocol, focusing on Apache Arrow RecordBatch construction, RecordBatchWriter, and PostgreSQL's SPI_getbinval(). Determine whether copying into RecordBatch or SPI_getbinval()/nocachegetattr() dominates, then measure the completed change against the PostgreSQL protocol baseline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, postgresql
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100