[Bug] Map with add_length will cause memory OOM
Open
- Dominant language
- Python
- Stars
- 5.2k
- Forks
- 448
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 26
Description
I use xtuner to handle the custom dataset, 200W items, num_proc=32. It will consume more than 1T memory.
This issue caused by
https://github.com/InternLM/xtuner/blob/49104763f69a4ac8971d6545ca3c1d1b6f965706/xtuner/dataset/huggingface.py#L216
Similar question: https://discuss.huggingface.co/t/multiprocessing-map-taking-too-much-memory-footprint/27238/17
Solution: Use num_proc=1 for add_length map operation.
Contributor guide
Assessment
This issue has not been assessed yet.