xarray/tests/test_dask.py::TestToDaskDataFrame::test_to_dask_dataframe_* test failures when dask+pyarrow are installed
Open
Nobody has claimed this yet.
bug
needs triage
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened?
When running the test suite in an environment where both dask and pyarrow are installed, two tests fail (log below):
FAILED xarray/tests/test_dask.py::TestToDaskDataFrame::test_to_dask_dataframe_2D - AssertionError: Attributes of DataFrame.iloc[:, 1] (column name="y") are different
FAILED xarray/tests/test_dask.py::TestToDaskDataFrame::test_to_dask_dataframe_not_daskarray - AssertionError: DataFrame.index are different
What did you expect to happen?
Tests passing ;-).
Minimal Complete Verifiable Example
pip install . pytest 'dask[complete]'
python -m pytest xarray/tests/test_dask.py
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
Relevant log output
____________________________________________ TestToDaskDataFrame.test_to_dask_dataframe_2D ____________________________________________
self = <xarray.tests.test_dask.TestToDaskDataFrame object at 0x7f59a87bce50>
def test_to_dask_dataframe_2D(self):
# Test if 2-D dataset is supplied
w = np.random.randn(2, 3)
ds = Dataset({"w": (("x", "y"), da.from_array(w, chunks=(1, 2)))})
ds["x"] = ("x", np.array([0, 1], np.int64))
ds["y"] = ("y", list("abc"))
# dask dataframes do not (yet) support multiindex,
# but when it does, this would be the expected index:
exp_index = pd.MultiIndex.from_arrays(
[[0, 0, 0, 1, 1, 1], ["a", "b", "c", "a", "b", "c"]], names=["x", "y"]
)
expected = pd.DataFrame({"w": w.reshape(-1)}, index=exp_index)
# so for now, reset the index
expected = expected.reset_index(drop=False)
actual = ds.to_dask_dataframe(set_index=False)
assert isinstance(actual, dd.DataFrame)
> assert_frame_equal(expected, actual.compute())
E AssertionError: Attributes of DataFrame.iloc[:, 1] (column name="y") are different
E
E Attribute "dtype" are different
E [left]: object
E [right]: string[pyarrow]
/tmp/xarray/xarray/tests/test_dask.py:822: AssertionError
______________________________________ TestToDaskDataFrame.test_to_dask_dataframe_not_daskarray _______________________________________
self = <xarray.tests.test_dask.TestToDaskDataFrame object at 0x7f59a87d5750>
def test_to_dask_dataframe_not_daskarray(self):
# Test if DataArray is not a dask array
x = np.random.randn(10)
y = np.arange(10, dtype="uint8")
t = list("abcdefghij")
ds = Dataset({"a": ("t", x), "b": ("t", y), "t": ("t", t)})
expected = pd.DataFrame({"a": x, "b": y}, index=pd.Index(t, name="t"))
actual = ds.to_dask_dataframe(set_index=True)
assert isinstance(actual, dd.DataFrame)
> assert_frame_equal(expected, actual.compute())
/tmp/xarray/xarray/tests/test_dask.py:867:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
left = Index(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'], dtype='object', name='t')
right = Index(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'], dtype='string', name='t'), obj = 'DataFrame.index'
def _check_types(left, right, obj: str = "Index") -> None:
if not exact:
return
assert_class_equal(left, right, exact=exact, obj=obj)
assert_attr_equal("inferred_type", left, right, obj=obj)
# Skip exact dtype checking when `check_categorical` is False
if is_categorical_dtype(left.dtype) and is_categorical_dtype(right.dtype):
if check_categorical:
assert_attr_equal("dtype", left, right, obj=obj)
assert_index_equal(left.categories, right.categories, exact=exact)
return
> assert_attr_equal("dtype", left, right, obj=obj)
E AssertionError: DataFrame.index are different
E
E Attribute "dtype" are different
E [left]: object
E [right]: string[pyarrow]
/tmp/xarray/.venv/lib/python3.11/site-packages/pandas/_testing/asserters.py:250: AssertionError
Anything else we need to know?
No response
Environment
INSTALLED VERSIONS
------------------
commit: 83c2919b27b4b2d8a01bfa380226134c71321aa0
python: 3.11.4 (main, Jun 8 2023, 06:01:19) [GCC 13.1.1 20230527]
python-bits: 64
OS: Linux
OS-release: 6.4.7-gentoo-dist
machine: x86_64
processor: AMD Ryzen 5 3600 6-Core Processor
byteorder: little
LC_ALL: None
LANG: pl_PL.UTF-8
LOCALE: ('pl_PL', 'UTF-8')
libhdf5: None
libnetcdf: None
xarray: 2023.8.0
pandas: 2.0.3
numpy: 1.25.2
scipy: None
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
iris: None
bottleneck: None
dask: 2023.8.1
distributed: 2023.8.1
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2023.6.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 68.1.2
pip: 23.2.1
conda: None
pytest: 7.4.0
mypy: None
IPython: None
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with xarray/tests/test_dask.py, especially TestToDaskDataFrame::test_to_dask_dataframe_2D and test_to_dask_dataframe_not_daskarray. Reproduce the failures using pytest with dask and pyarrow installed, then trace the dataframe conversion and its dtype expectations. Done means both tests pass without the reported string dtype and index mismatches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100