Explode method does not work for object column with DatetimeInterval values
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the issue**:
It seems that dask `explode()` method does not work with non-list values and throws unclear error. However, there is a comment in the code that in case of non-list values it fallbacks to pandas implementation but it fails.
**Minimal Complete Verifiable Example**:
Pandas woking example:
```python
import pandas as pd
df = (pd.DataFrame({
'A': [1, 2, 3],
'start': [pd.to_datetime('2024-01-01'), pd.to_datetime('2024-02-01'), pd.to_datetime('2024-03-01')],
'end': [pd.to_datetime('2024-01-3'), pd.to_datetime('2024-02-3'), pd.to_datetime('2024-03-3')]})
.assign(DateRange= lambda x: x.apply(lambda row: pd.date_range(start=row['start'], end=row['end'], freq='D'), axis=1)))
display(df)
df.explode('DateRange')
```
Dask failing example:
```python
from dask.distributed import LocalCluster
import dask.dataframe as dd
cluster = LocalCluster() # Fully-featured local Dask cluster
client = cluster.get_client()
df = dd.from_pandas(pd.DataFrame({
'A': [1, 2, 3],
'start': [pd.to_datetime('2024-01-01'), pd.to_datetime('2024-02-01'), pd.to_datetime('2024-03-01')],
'end': [pd.to_datetime('2024-01-3'), pd.to_datetime('2024-02-3'), pd.to_datetime('2024-03-3')]}), npartitions=2)
(df.assign(DateRange= lambda x: x.apply(lambda row: pd.date_range(start=row['start'], end=row['end'], freq='D'), axis=1))
.explode('DateRange'))
```
```text
{
"name": "AttributeError",
"message": "'StringDtype' object has no attribute 'pyarrow_dtype'",
"stack": "---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[49], line 13
4 client = cluster.get_client()
6 df = dd.from_pandas(pd.DataFrame({
7 'A': [1, 2, 3],
8 'start': [pd.to_datetime('2024-01-01'), pd.to_datetime('2024-02-01'), pd.to_datetime('2024-03-01')],
9 'end': [pd.to_datetime('2024-01-3'), pd.to_datetime('2024-02-3'), pd.to_datetime('2024-03-3')]}), npartitions=2)
12 (df.assign(DateRange= lambda x: x.apply(lambda row: pd.date_range(start=row['start'], end=row['end'], freq='D'), axis=1))
---> 13 .explode('DateRange'))
File ~/src/demand-forecasting/.venv/lib/python3.10/site-packages/dask/dataframe/core.py:5754, in DataFrame.explode(self, column)
5752 @derived_from(pd.DataFrame)
5753 def explode(self, column):
-> 5754 meta = self._meta.explode(column)
5755 return self.map_partitions(M.explode, column, meta=meta, enforce_metadata=False)
File ~/src/demand-forecasting/.venv/lib/python3.10/site-packages/pandas/core/frame.py:9819, in DataFrame.explode(self, column, ignore_index)
9817 df = self.reset_index(drop=True)
9818 if len(columns) == 1:
-> 9819 result = df[columns[0]].explode()
9820 else:
9821 mylen = lambda x: len(x) if (is_list_like(x) and len(x) > 0) else 1
File ~/src/demand-forecasting/.venv/lib/python3.10/site-packages/pandas/core/series.py:4530, in Series.explode(self, ignore_index)
4477 \"\"\"
4478 Transform each element of a list-like to a row.
4479
(...)
4527 dtype: object
4528 \"\"\"
4529 if isinstance(self.dtype, ExtensionDtype):
-> 4530 values, counts = self._values._explode()
4531 elif len(self) and is_object_dtype(self.dtype):
4532 values, counts = reshape.explode(np.asarray(self._values))
File ~/src/demand-forecasting/.venv/lib/python3.10/site-packages/pandas/core/arrays/arrow/array.py:1781, in ArrowExtensionArray._explode(self)
1776 \"\"\"
1777 See Series.explode.__doc__.
1778 \"\"\"
1779 # child class explode method supports only list types; return
1780 # default implementation for non list types.
-> 1781 if not pa.types.is_list(self.dtype.pyarrow_dtype):
1782 return super()._explode()
1783 values = self
AttributeError: 'StringDtype' object has no attribute 'pyarrow_dtype'"
}
```
**Environment**:
- Dask version: '2024.1.0'
- Python version: 3.10
- Pandas: 2.2.0
- Operating System: Linux, Ubuntu
- Install method (conda, pip, source): poetry
Contributor guide
Assessment
This issue has not been assessed yet.