huggingface / huggingface/datasets
Adding column with dict struction when mapping lead to wrong order
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
in `map()` function, I want to add a new column with a dict structure.
```
def map_fn(example):
example['text'] = {'user': ..., 'assistant': ...}
return example
```
However this leads to a wrong order `{'assistant':..., 'user':...}` in the dataset.
Thus I can't concatenate two datasets due to the different feature structures.
[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example
This seems an issue in low level pyarrow library instead of datasets, however, I think datasets should allow concatenate two datasets actually in the same structure.
### Steps to reproduce the bug
[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example
### Expected behavior
two datasets could be concatenated.
### Environment info
N/A
Contributor guide
Assessment
This issue has not been assessed yet.