huggingface / huggingface/datasets
[FR] `map` should reuse unchanged columns from the previous dataset to avoid disk usage
Open
enhancement
good second issue
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Feature request
Currently adding a new column with `map` will cause all the data in the dataset to be duplicated and stored/cached on the disk again. It should reuse unchanged columns.
### Motivation
This allows having datasets with different columns but sharing some basic columns. Currently, these datasets would become too expensive to store and one would need some kind of on-the-fly join; which also doesn't seem implemented.
### Your contribution
_
Contributor guide
Assessment
This issue has not been assessed yet.