dask / dask/dask-image

New Dask Arrow-based strings cause test failures

Open
#335 14 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
226
Forks
56
PR merge metrics
No merged PRs in 30d

Description

(Edited by @m-albert)

In the presence of `pyarrow`, dask by default assumes dataframes of type object to be pyarrow strings (see https://github.com/dask/dask/issues/10139#issuecomment-1655928619).

This creates problems revealed by failing tests (e.g. `test_dask_image/test_ndmeasure/test_find_objects.py::test_3d_find_objects`)

https://github.com/dask/dask-image/blob/67540af25597f84e4a642d644ba30dce7aebe753/dask_image/ndmeasure/_utils/_find_objects.py#L68-L70

`dd.from_delayed(df1, meta=meta).compute().dtypes`

Working install:
> 0 object
1 object
2 object
dtype: object

Failing install:
>0 string[pyarrow]
1 string[pyarrow]
2 string[pyarrow]
dtype: object

The failing test had come up when releasing v2023.08.0 in https://github.com/conda-forge/dask-image-feedstock/pull/14.

@jakirkham found that `pyarrow` is installed with the [conda distribution of dask](https://github.com/conda-forge/dask-feedstock/blob/89ff1ea9116e0798d9073d819a6b9b22fa1d0917/recipe/meta.yaml#L31), but not when [installing over pip](https://github.com/dask/dask/blob/main/pyproject.toml), where it just part of the `[complete]` target.

Also @jakirkham found that the above described conflicting behaviour can be [turned off using the dask configuration](https://github.com/dask-contrib/dask-sql/commit/af180620216793b7c6125b7db8f98dd0bba038de#diff-a31c7ed5d35f5ed8233994868c54d625b18e6bacb6794344c4531e62bd9dde59R15-R16).

He did this for the tests performed by the dask-image conda feedstock on [v2023.08.0](https://github.com/conda-forge/dask-image-feedstock/pull/14).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.