apache / apache/arrow-adbc

c/driver/postgresql: Handle non-COPY path results where individual string data would be larger than 2GB

Open
#2,064 0 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
C#
Stars
627
Forks
217
Avg merge
17h
Merged PRs (30d)
57

Description

### What feature or improvement would you like to see?

In the COPY reader we go to quite some lengths to ensure that we don't overflow the 2GB limit for the Arrow string type:

https://github.com/apache/arrow-adbc/blob/45cd9be77a619fe9b42f6b6470010ad430e5a868/c/driver/postgresql/statement.cc#L770-L776

This was not implemented in the PR that enabled converting PQresult objects to arrays (https://github.com/apache/arrow-adbc/pull/2029). While it is less likely that one of these queries will overflow (given that it is opt-in), we have the infrastructure to handle it and should!

Contributor guide

Open the contributing guide

Research direction

Start in c/driver/postgresql/statement.cc at lines 770-776, then trace the PQresult-to-array conversion added by PR #2029 and compare it with the COPY reader's handling. Done means non-COPY results safely handle an individual string larger than 2GB without overflowing the Arrow string limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, postgresql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.