apache / apache/arrow-adbc

## Add Server-Side Cursor/Streaming Support to `adbc_driver_postgresql.dbapi`

Open
#2,656 0 comments 2 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
C#
Stars
627
Forks
217
Avg merge
17h
Merged PRs (30d)
57

Description

### What feature or improvement would you like to see?

## Add Server-Side Cursor/Streaming Support to `adbc_driver_postgresql.dbapi`

Maybe I'm missing soemthing, but I’m hitting memory issues with large PostgreSQL queries in `adbc_driver_postgresql.dbapi` because it fetches full result sets upfront. PostgreSQL supports server-side cursors (`DECLARE CURSOR`, `FETCH`) for chunked retrieval, but the driver doesn’t use them natively. I’d like streaming or cursor support to process big results as Arrow tables in chunks, avoiding `LIMIT/OFFSET`.

Current behavior:
```python
with adbc_driver_postgresql.dbapi.connect(uri) as conn:
cursor = conn.cursor()
cursor.execute("SELECT * FROM huge_table")
chunk = cursor.fetchmany(1000) # Still loads all rows into memory first
```
Request: Add a `chunk_size` param or `fetch_arrow_stream()` to fetch results incrementally, e.g.:
```
cursor.execute("SELECT * FROM huge_table", chunk_size=1000)
for chunk in cursor.fetch_arrow_stream():
print(chunk.num_rows) # 1000 rows at a time
```
Could leverage `libpq` cursors or single-row mode. I'm guessing workaround is manual SQL cursors, but that’s clunky.

Contributor guide

Open the contributing guide

Research direction

Start by locating the adbc_driver_postgresql.dbapi cursor implementation and tracing execute() with fetchmany(); the issue does not name a file or test. Compare the proposed chunk_size and fetch_arrow_stream() behaviors, then define completion as incremental Arrow chunks without loading the full PostgreSQL result into memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.