feat: implement true server-side cursors for .stream()
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 13
- Forks
- 5
- Avg merge
- 1h 49m
- Merged PRs (30d)
- 3
Description
The current .stream() implementation relies on LIMIT/OFFSET chunking. While effective, this becomes slow for very large offsets and puts pressure on the database to re-scan results.
Goal: Implement true server-side cursors (using DECLARE CURSOR in Postgres/MySQL) to allow efficient streaming of millions of rows.
Implementation hint:
- Update the
ryx-backendexecutor to support named cursors. - Modify the
_stream_querysetgenerator in Python to fetch from the cursor in chunks instead of issuing newLIMIT/OFFSETqueries. - Handle the transaction requirement (cursors usually require an open transaction).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the current _stream_queryset generator and the ryx-backend executor that performs its LIMIT/OFFSET queries. Determine how named cursors and their required open transaction would fit the existing flow; done means streaming in chunks through server-side cursors for Postgres/MySQL without issuing new offset queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, postgres, python, rust
- Domain
- backend, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100