snowflakedb / snowflakedb/snowpark-python
SNOW-704049: Pearson Correlation, Covariance
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 341
- Forks
- 155
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 27
Description
What is the current behavior?
SQL %CORR or SQL %COVAR is called via DataFrame.stat.corr(). It returns single value instead of a correlation or covariance matrix with all pair-wise coefficients. The procedure is not mature and doesn't scale to wide tables.
What is the desired behavior?
Return symmetrical output matrix with all correlation coefficients with the ability to run on wide 1k + columns. Pandas.corr(), R corr(method=Pearson), SAS proc corr all do this Snowflake/Snowpark does not.
How would this improve snowflake-snowpark-python?
It would provide additional functionality for a core Feature Engineering/DS tool. It would scale to wide tables where Corr is used. It would meet customer demand(Fidelity, GIC).
References, Other Background
This topic has been widely discussed for a long time. See - https://snowflakecomputing.atlassian.net/browse/SNOW-587357.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the DataFrame.stat.corr() entry point and review the SQL %CORR/%COVAR behavior described in the issue. Use the linked SNOW-587357 discussion for background; done means returning a symmetrical matrix of all pair-wise correlation or covariance coefficients and supporting tables with 1,000+ columns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- data, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100