dd.concat crashes with unhelpful error message when column types are incompatible
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
# Repro
Run:
```
import dask.dataframe as dd
import pandas as pd
ddf1 = dd.from_pandas(pd.DataFrame([{'foo': float('nan')}]), npartitions=1)
ddf2 = dd.from_pandas(pd.DataFrame([{'foo': ['string']}]), npartitions=1)
ddf2['foo'] = ddf2['foo'].astype('category')
dd.concat([ddf1, ddf2])
```
You will get the following error:
```
Traceback (most recent call last)
[](https://localhost:8080/#) in ()
----> 1 dd.concat([ddf1, ddf2])
5 frames
[/usr/local/lib/python3.7/dist-packages/pandas/core/dtypes/concat.py](https://localhost:8080/#) in (.0)
272 if not all(
273 is_dtype_equal(other.categories.dtype, first.categories.dtype)
--> 274 for other in to_union[1:]
275 ):
276 raise TypeError("dtype of categories must be the same")
AttributeError: 'numpy.ndarray' object has no attribute 'categories'
```
# Why is this painful?
The error gives no indication that there's a column type mismatch. In the actual situation where I encountered this error, the dataframes were much more complex, so I initially had no suspicion that the column types were the issue. I had to spend about 20 minutes poking around before I finally realized what was going on.
Contributor guide
Assessment
This issue has not been assessed yet.