cloudflare / cloudflare/sqlalchemy-clickhouse

Is there a way to stream large result set

Open
#26 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
325
Forks
101
PR merge metrics
No merged PRs in 30d

Description

Currently the `Cursor.execute` will retrieve all the result rows into memory, this could cause memory and performance issues if the result rows is very large (e.g. `SELECT * FROM ontime`).

Is there a better solution for this situation?

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the Cursor.execute entry point mentioned in the issue and trace how result rows are retrieved and stored. Compare the available SQLAlchemy result-streaming patterns and existing tests, if any. Done means large queries can be consumed without loading every row into memory, with behavior covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlalchemy
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.