SNOW-1358494: Pandas Replace on Copy of DataFrame has No Effect

Please answer these questions before submitting your issue. Thanks!

1. What version of Python are you using?

3.11.5

2. What operating system and processor architecture are you using?

macOS-14.4.1-arm64-arm-64bit

3. What are the component versions in the environment (pip freeze)?

annotated-types==0.6.0
asn1crypto==1.5.1
black==24.4.0
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==2.2.1
coverage==7.5.0
cryptography==42.0.5
filelock==3.13.4
flake8==7.0.0
idna==3.7
iniconfig==2.0.0
isort==5.13.2
mccabe==0.7.0
mypy-extensions==1.0.0
numpy==1.26.4
packaging==24.0
pandas==2.2.2
pathspec==0.12.1
platformdirs==4.2.0
pluggy==1.4.0
pyarrow==15.0.2
pycodestyle==2.11.1
pycparser==2.22
pydantic==2.5.3
pydantic_core==2.14.6
pyflakes==3.2.0
PyJWT==2.8.0
pyOpenSSL==24.1.0
pytest==8.1.1
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
requests==2.31.0
six==1.16.0
snowflake-connector-python==3.8.1
snowflake-snowpark-python==1.15.0
sortedcontainers==2.4.0
tomlkit==0.12.4
typing_extensions==4.11.0
tzdata==2024.1
urllib3==2.2.1

4. What did you do?

Using a mock snowpark session for in memory unit testing we receive warnings for improper actions on a pandas copy of a dataframe.

import warnings
import numpy as np
from snowflake.snowpark import Session
import pytest

@pytest.fixture()
def snowpark_session() -> Session:
    return Session.builder.config("local_testing", True).create()

def test_pandas_error(snowpark_session: Session):
    with warnings.catch_warnings():
        warnings.simplefilter("error")
        snowpark_session.create_dataframe(
            [{"a": 1.1, "b": "s"}, {"a": np.nan, "b": "t"}],
            schema=["a", "b"]
        )

During my tests I receive the following warning.

snowpark/mock/_plan.py:565: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
  The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.
  
  For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.

The contents of the lines in snowpar/mock/_plan:565 are as follows.

        for column_name in table.columns:
            sf_type = table.sf_types[column_name]
            table[column_name].sf_type = table.sf_types[column_name]
            if not isinstance(sf_type.datatype, _NumericType):
                table[column_name].replace(np.nan, None, inplace=True)

As you can see by inspection, the in place replace method is executed on a slice of a DataFrame, which is a copy in Pandas. If the code passes tests now and works, then this whole block of code really is not needed and raises an unnecessary warning. It appears that someone was trying to replace np.nan with None only on numeric columns. We can delete this code and silence the warning while preventing future confusion when people read this code and think that np.nan will be replaced with None.

5. What did you expect to see?

This is a useful warning from pandas. I expect to not have code in the codebase which has no effect or if the effect is desired I expect the code to be fixed.

6. Can you set logging to DEBUG and collect the logs?

See above for the relevant sections.

Start at snowpark/mock/_plan.py around line 565 and inspect the DataFrame column replacement block. Reproduce the warning with the provided mock Session test, then verify the relevant behavior after the change. Done means the pandas warning is no longer emitted and the intended handling of NaN values is confirmed.