huggingface / huggingface/datasets
flatten_indices doesn't work with pandas format
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
Hi,
I noticed that `flatten_indices` throws an error when the batch format is `pandas`. This is probably due to the fact that flatten_indices uses map internally which doesn't accept dataframes as the transformation function output
### Steps to reproduce the bug
tabular_data = pd.DataFrame(np.random.randn(10,10))
tabular_data = datasets.arrow_dataset.Dataset.from_pandas(tabular_data)
tabular_data.with_format("pandas").select([0,1,2,3]).flatten_indices()
### Expected behavior
No error thrown
### Environment info
- `datasets` version: 2.10.1
- Python version: 3.9.5
- PyArrow version: 11.0.0
- Pandas version: 1.4.1
Contributor guide
Assessment
This issue has not been assessed yet.