snowflakedb / snowflakedb/snowpark-python
SNOW-704114: Implement `DataFrame.summary` to show percentiles
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 341
- Forks
- 155
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 27
Description
Update: As of Mar.27 2023, @sfc-gh-stan and @sfc-gh-gfrere discussed this and concluded we should implement DataFrame.summary() to display the percentiles instead of modifying the existing DataFrame.describe().
What is the current behavior?
Example::
>>> df = session.create_dataframe([[1, 2], [3, 4]], schema=["a", "b"])
>>> desc_result = df.describe().sort("SUMMARY").show()
-------------------------------------------------------
|"SUMMARY" |"A" |"B" |
-------------------------------------------------------
|count |2.0 |2.0 |
|max |3.0 |4.0 |
|mean |2.0 |3.0 |
|min |1.0 |2.0 |
|stddev |1.4142135623730951 |1.4142135623730951 |
-------------------------------------------------------
Percentiles are not shown.
What is the desired behavior?
Show percentiles 25, 50, 75 like Pandas.describe(). Using updated approximate percentile
'alter session set APPROX_PERCENTILE_EXACT_IF_POSSIBLE = true;’
How would this improve snowflake-snowpark-python?
It would match Pandas behavior and is requested by customers. Given the improvements being made to APPROX_PERCENTILE% we should implement this change. Before we may have thought this was too expensive or inaccurate due to APPROX_PERCENTILE% issues on few samples.
References, Other Background
APPROX_PERCENTILE is being updated by the SQL Compiler team to allow for a larger buffer usage and if the buffer doesn't fill PERCENTILE_CONT is automatically used.
https://snowflakecomputing.atlassian.net/browse/SNOW-704037
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the DataFrame.summary() and existing DataFrame.describe() entry points in the snowpark-python codebase, then inspect how summary statistics are assembled. Verify the expected percentile behavior against the provided example and Pandas.describe(), including the requested approximate-percentile session setting. Done means DataFrame.summary() displays the 25th, 50th, and 75th percentiles without changing DataFrame.describe().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100