llama-factory 0.9.4 dev0:TypeError when running full sft
- Dominant language
- No language data
- Stars
- 861
- Forks
- 51
- Avg merge
- 1m
- Merged PRs (30d)
- 1
Description
Error
```shell
Running tokenizer on dataset (num_proc=16): 0%| | 0/1000 [00:00
[rank0]: launch()
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/launcher.py", line 19, in launch
[rank0]: run_exp()
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/train/tuner.py", line 110, in run_exp
[rank0]: _training_function(config={"args": args, "callbacks": callbacks})
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/train/tuner.py", line 72, in _training_function
[rank0]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 51, in run_sft
[rank0]: dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/data/loader.py", line 315, in get_dataset
[rank0]: dataset = _get_preprocessed_dataset(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/data/loader.py", line 256, in _get_preprocessed_dataset
[rank0]: dataset = dataset.map(
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
[rank0]: out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3171, in map
[rank0]: for rank, done, content in iflatmap_unordered(
[rank0]: File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/utils/py_utils.py", line 721, in iflatmap_unordered
[rank0]: raise RuntimeError(
[rank0]: RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing.
[rank0]:[W826 17:05:49.980127574 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.