## Add Server-Side Cursor/Streaming Support to `adbc_driver_postgresql.dbapi`
- Dominant language
- C#
- Stars
- 627
- Forks
- 217
- Avg merge
- 17h
- Merged PRs (30d)
- 57
Description
### What feature or improvement would you like to see?
## Add Server-Side Cursor/Streaming Support to `adbc_driver_postgresql.dbapi`
Maybe I'm missing soemthing, but I’m hitting memory issues with large PostgreSQL queries in `adbc_driver_postgresql.dbapi` because it fetches full result sets upfront. PostgreSQL supports server-side cursors (`DECLARE CURSOR`, `FETCH`) for chunked retrieval, but the driver doesn’t use them natively. I’d like streaming or cursor support to process big results as Arrow tables in chunks, avoiding `LIMIT/OFFSET`.
Current behavior:
```python
with adbc_driver_postgresql.dbapi.connect(uri) as conn:
cursor = conn.cursor()
cursor.execute("SELECT * FROM huge_table")
chunk = cursor.fetchmany(1000) # Still loads all rows into memory first
```
Request: Add a `chunk_size` param or `fetch_arrow_stream()` to fetch results incrementally, e.g.:
```
cursor.execute("SELECT * FROM huge_table", chunk_size=1000)
for chunk in cursor.fetch_arrow_stream():
print(chunk.num_rows) # 1000 rows at a time
```
Could leverage `libpq` cursors or single-row mode. I'm guessing workaround is manual SQL cursors, but that’s clunky.
Contributor guide
Research direction
Start by locating the adbc_driver_postgresql.dbapi cursor implementation and tracing execute() with fetchmany(); the issue does not name a file or test. Compare the proposed chunk_size and fetch_arrow_stream() behaviors, then define completion as incremental Arrow chunks without loading the full PostgreSQL result into memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100