snowflakedb / snowflakedb/snowpark-python
SNOW-902662: DF.to_pandas_batches() batch size parameter
Open
@sfc-gh-ashahi is already working on this.
Since Mar 20, 2024.
feature
status-triage_done
- Dominant language
- Python
- Stars
- 341
- Forks
- 155
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 27
Description
Current behaviour
When using DataFrame.to_pandas_batches() it returns a Pandas Dataframe Iterator that generates Pandas Dataframes with a "random" number of rows.
Desired behaviour
I would like to insert a parameter in the to_pandas_batches() method, where I fix the number of rows for each Pandas Dataframe generated.
How would this improve snowflake-snowpark-python?
This would be beneficial since the user would be able to control the chunk sizes to process and be sure that his processes don't get overloaded by the snowflake back-end's calculation of the number of rows.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.