snowflakedb / snowflakedb/snowpark-python
SNOW-1889503: [Local Testing] Group by in empty dataframe adds "phantom" row with NULL group by key
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 341
- Forks
- 155
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 27
Description
Please answer these questions before submitting your issue. Thanks!
- What version of Python are you using?
3.11.11
- What operating system and processor architecture are you using?
Linux-6.8.0-51-generic-x86_64-with-glibc2.39
- What are the component versions in the environment (
pip freeze)?
N/A
-
What did you do?
Performing a group by on an empty dataframe produces a dataframe with a row where the key is NULL.
Simple reproducible example:
from snowflake.snowpark import Session
from snowflake.snowpark.functions import when_matched, when_not_matched, col, sum
session = Session.builder.config('local_testing', True).create()
df = session.create_dataframe([[1, 5]], schema=['id', 'count'])
df.filter(col("id") > 1).group_by(col("id")).agg(sum(col("count")).alias("count")).show()
-
What did you expect to see?
An empty DataFrame should be shown.
However, we obtain the following output:
------------------
|"ID" |"COUNT" |
------------------
|NULL |nan |
------------------
- Can you set logging to DEBUG and collect the logs?
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the supplied Python reproducer with local_testing enabled, focusing on the group_by and agg operations on the filtered empty dataframe. Done means the result is an empty DataFrame rather than a row containing a NULL group key and NaN aggregate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100