snowflakedb / snowflakedb/snowpark-python

SNOW-1242819: `to_pandas_batches` function Not working with address-like strings in string column

Open
#1,319 3 comments 0 reactions 1 assignee View on GitHub

@sfc-gh-ashahi is already working on this.

Since Mar 20, 2024.

bug needs triage
Dominant language
Python
Stars
341
Forks
155
Avg merge
4d 16h
Merged PRs (30d)
27

Description

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using?

Python 3.10.7

  1. What operating system and processor architecture are you using?

linux 64

  1. What are the component versions in the environment (pip freeze)?

snowflake-snowpark-python ==1.13.0

  1. What did you do?
    So I basically have a table on snowflake that has an address-like field(ie like the string "308 Negra Arroyo Lane" off of breaking bad). Using snowpark, I pass this table through a pipeline that runs in a docker container for processing(and yes it needs to be in csv form hence the pandas middle man). Originally, I used the to_pandas method and it was just fine, as the resulting pandas dataframe did not try to read this column in as a string. However this table is large enough to where I need to run to_pandas_batches to prevent the whole thing from loading into memory via Pandas. However for dataframe batches outside of the first, pandas is attempting to process this column as a numeric type(which it clearly isn't). So for example lets say the first patch dataframe has that walter white address column and the second one also has that address, the second dataframe will throw an error as it tries to parse that column as a number.

  2. What did you expect to see?

I expected the resulting dataframe to behave like to_pandas where the pandas dataframe had no issue recognizing the address-like column as a string and did not try to do a numeric conversion. What actually happened was the pandas dataframe attempted to read this column in as a number. There probably needs to be a tweak made to _fix_pandas_df_fixed_type to factor in this edge case going forward of a string that starts with a numeric char.

  1. Can you set logging to DEBUG and collect the logs?
{Omitting Logs above this one that confirm the pipeline was successful for the first batch of address like strings}

INFO -     for df in address_df.to_pandas_batches():
env/lib/python3.10/site-packages/snowflake/snowpark/_internal/server_connection.py", line 731, in _fix_pandas_df_fixed_type
 INFO -     pd_df[pandas_col_name] = pandas.to_numeric(
INFO -   File "/env/lib/python3.10/site-packages/pandas/core/tools/numeric.py", line 185, in to_numeric
 INFO -     values, _ = lib.maybe_convert_numeric(
INFO -   File "pandas/_libs/lib.pyx", line 2411, in pandas._libs.lib.maybe_convert_numeric
INFO - ValueError: Unable to parse string "123 FAKE ADDRESS" at position 0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.