snowflakedb / snowflakedb/snowpark-python
SNOW-944048: Support Polars with a to_polars method
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 341
- Forks
- 155
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 27
Description
What is the current behavior?
Currently working with data happens either really fast in Snowflake with the help of Snowpark Dataframes or super slow and on a single core when transformed to Pandas with to_pandas(). This can be especially painful for UDFs.
What is the desired behavior?
Also support polars dataframes. Polars is blazingly fast, multi-threaded and makes use of all cores on a node. It's also gaining a lot of traction, 20.9k Github stars compared to 40k of Pandas. It's already used used in real-world project and we see dev teams migrating from Pandas to Polars. In the end it would be nice to have a to_polars() dataframe method.
How would this improve snowflake-snowpark-python?
It would allow UDFs to be much faster for custom code in cases one would need to refrain to to_pandas normally.
References, Other Background
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing dataframe conversion path for to_pandas() and the surrounding Snowpark DataFrame API. Investigate how Polars should be integrated for UDF workflows, including dependency and conversion boundaries. Done means a supported to_polars() method with defined behavior and coverage for the new conversion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100