map_partitions doesn't work with Scalars
Open
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
dask/dask works here, not sure how important this is though.
reproducer:
```
import pandas as pd
pdf = pd.DataFrame({"a": [1, 2]})
df = from_pandas(npartitions=2)
map_partitions(lambda x: x, df.a.sum())
```
Contributor guide
Research direction
Start by running the pandas reproducer involving from_pandas, df.a.sum(), and map_partitions(lambda x: x, ...). Compare its behavior with dask/dask, as the issue reports that implementation works there. Done means the scalar case behaves consistently, with a regression test covering the reproducer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100