InternLM / InternLM/Intern-S1

llama-factory 0.9.4 dev0:TypeError when running full sft

Open
#22 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
861
Forks
51
Avg merge
1m
Merged PRs (30d)
1

Description

Error
```shell
Running tokenizer on dataset (num_proc=16): 0%| | 0/1000 [00:00
[rank0]: launch()
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/launcher.py", line 19, in launch
[rank0]: run_exp()
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/train/tuner.py", line 110, in run_exp
[rank0]: _training_function(config={"args": args, "callbacks": callbacks})
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/train/tuner.py", line 72, in _training_function
[rank0]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 51, in run_sft
[rank0]: dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/data/loader.py", line 315, in get_dataset
[rank0]: dataset = _get_preprocessed_dataset(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/data/loader.py", line 256, in _get_preprocessed_dataset
[rank0]: dataset = dataset.map(
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
[rank0]: out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3171, in map
[rank0]: for rank, done, content in iflatmap_unordered(
[rank0]: File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/utils/py_utils.py", line 721, in iflatmap_unordered
[rank0]: raise RuntimeError(
[rank0]: RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing.
[rank0]:[W826 17:05:49.980127574 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.