snowflakedb / snowflakedb/snowpark-python
SNOW-1797580: Integer columns contain Na after filtering when using to pandas in local testing
Open
Nobody has claimed this yet.
bug
status-triage_done
- Dominant language
- Python
- Stars
- 341
- Forks
- 155
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 27
Description
-
What version of Python are you using?
Python 3.11.8
-
What operating system and processor architecture are you using?
Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
-
What are the component versions in the environment (
pip freeze)?snowflake-connector-python==3.12.3
snowflake-snowpark-python==1.24.0 -
What did you do?
if __name__ == "__main__":
from snowflake.snowpark import Session
import snowflake.snowpark.functions as spf
conn_params = {
"schema": "SCHEMA",
"local_testing": True,
}
session = Session.builder.configs(conn_params).create()
data = [
[1, False],
[1, False],
[1, False],
[2, True],
]
schema = ["INT_COL", "BOOL_COL"]
df = session.create_dataframe(data, schema)
df = df.with_column("INT_COL", spf.cast("INT_COL", "int"))
filtered = df.filter(
spf.col("BOOL_COL")
)
pd_df = filtered.to_pandas()
collected = filtered.collect()
-
What did you expect to see?
That the pd_df has the same data as collected. But the int column is NaN for the pandas df. I already found the issue and will open a PR asap
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.