dask / dask/dask

dd.concat crashes with unhelpful error message when column types are incompatible

Open
#9,079 2 comments 2 reactions 0 assignees View on GitHub
bug dataframe
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

# Repro

Run:

```
import dask.dataframe as dd
import pandas as pd

ddf1 = dd.from_pandas(pd.DataFrame([{'foo': float('nan')}]), npartitions=1)
ddf2 = dd.from_pandas(pd.DataFrame([{'foo': ['string']}]), npartitions=1)
ddf2['foo'] = ddf2['foo'].astype('category')

dd.concat([ddf1, ddf2])
```

You will get the following error:

```
Traceback (most recent call last)
[](https://localhost:8080/#) in ()
----> 1 dd.concat([ddf1, ddf2])

5 frames
[/usr/local/lib/python3.7/dist-packages/pandas/core/dtypes/concat.py](https://localhost:8080/#) in (.0)
272 if not all(
273 is_dtype_equal(other.categories.dtype, first.categories.dtype)
--> 274 for other in to_union[1:]
275 ):
276 raise TypeError("dtype of categories must be the same")

AttributeError: 'numpy.ndarray' object has no attribute 'categories'
```

# Why is this painful?

The error gives no indication that there's a column type mismatch. In the actual situation where I encountered this error, the dataframes were much more complex, so I initially had no suspicion that the column types were the issue. I had to spend about 20 minutes poking around before I finally realized what was going on.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.