snowflakedb / snowflakedb/snowpark-python

SNOW-1889503: [Local Testing] Group by in empty dataframe adds "phantom" row with NULL group by key

Open
#2,886 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs triage
Dominant language
Python
Stars
341
Forks
155
Avg merge
4d 16h
Merged PRs (30d)
27

Description

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using?

3.11.11

  1. What operating system and processor architecture are you using?

Linux-6.8.0-51-generic-x86_64-with-glibc2.39

  1. What are the component versions in the environment (pip freeze)?

N/A

  1. What did you do?

    Performing a group by on an empty dataframe produces a dataframe with a row where the key is NULL.

Simple reproducible example:

from snowflake.snowpark import Session
from snowflake.snowpark.functions import when_matched, when_not_matched, col, sum

session = Session.builder.config('local_testing', True).create()

df = session.create_dataframe([[1, 5]], schema=['id', 'count'])

df.filter(col("id") > 1).group_by(col("id")).agg(sum(col("count")).alias("count")).show()
  1. What did you expect to see?

    An empty DataFrame should be shown.
    However, we obtain the following output:

------------------
|"ID"  |"COUNT"  |
------------------
|NULL  |nan      |
------------------
  1. Can you set logging to DEBUG and collect the logs?
    N/A

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied Python reproducer with local_testing enabled, focusing on the group_by and agg operations on the filtered empty dataframe. Done means the result is an empty DataFrame rather than a row containing a NULL group key and NaN aggregate.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.