snowflakedb / snowflakedb/snowpark-python
SNOW-1242819: `to_pandas_batches` function Not working with address-like strings in string column
@sfc-gh-ashahi is already working on this.
Since Mar 20, 2024.
- Dominant language
- Python
- Stars
- 341
- Forks
- 155
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 27
Description
Please answer these questions before submitting your issue. Thanks!
- What version of Python are you using?
Python 3.10.7
- What operating system and processor architecture are you using?
linux 64
- What are the component versions in the environment (
pip freeze)?
snowflake-snowpark-python ==1.13.0
-
What did you do?
So I basically have a table on snowflake that has an address-like field(ie like the string "308 Negra Arroyo Lane" off of breaking bad). Using snowpark, I pass this table through a pipeline that runs in a docker container for processing(and yes it needs to be in csv form hence the pandas middle man). Originally, I used theto_pandasmethod and it was just fine, as the resulting pandas dataframe did not try to read this column in as a string. However this table is large enough to where I need to runto_pandas_batchesto prevent the whole thing from loading into memory via Pandas. However for dataframe batches outside of the first, pandas is attempting to process this column as a numeric type(which it clearly isn't). So for example lets say the first patch dataframe has that walter white address column and the second one also has that address, the second dataframe will throw an error as it tries to parse that column as a number. -
What did you expect to see?
I expected the resulting dataframe to behave like to_pandas where the pandas dataframe had no issue recognizing the address-like column as a string and did not try to do a numeric conversion. What actually happened was the pandas dataframe attempted to read this column in as a number. There probably needs to be a tweak made to _fix_pandas_df_fixed_type to factor in this edge case going forward of a string that starts with a numeric char.
- Can you set logging to DEBUG and collect the logs?
{Omitting Logs above this one that confirm the pipeline was successful for the first batch of address like strings}
INFO - for df in address_df.to_pandas_batches():
env/lib/python3.10/site-packages/snowflake/snowpark/_internal/server_connection.py", line 731, in _fix_pandas_df_fixed_type
INFO - pd_df[pandas_col_name] = pandas.to_numeric(
INFO - File "/env/lib/python3.10/site-packages/pandas/core/tools/numeric.py", line 185, in to_numeric
INFO - values, _ = lib.maybe_convert_numeric(
INFO - File "pandas/_libs/lib.pyx", line 2411, in pandas._libs.lib.maybe_convert_numeric
INFO - ValueError: Unable to parse string "123 FAKE ADDRESS" at position 0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.