modelscope / modelscope/ms-swift
GKD 出现runtime error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
[INFO:swift] Memory cleanup attempt 1: Freed 0.00 GB reserved, 0.00 GB allocated
[rank2]: Traceback (most recent call last):
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/cli/rlhf.py", line 7, in
[rank2]: rlhf_main()
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank2]: return SwiftRLHF(args).main()
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/llm/base.py", line 49, in main
[rank2]: result = self.run()
[rank2]: ^^^^^^^^^^
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank2]: return func(self, *args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/llm/train/sft.py", line 196, in run
[rank2]: return self.train(trainer)
[rank2]: ^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/llm/train/sft.py", line 244, in train
[rank2]: trainer.train(trainer.args.resume_from_checkpoint)
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/trainers/mixin.py", line 816, in train
[rank2]: res = super().train(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/miniforge3/envs/ms-swift/lib/python3.11/site-packages/transformers/trainer.py", line 2206, in train
[rank2]: return inner_training_loop(
[rank2]: ^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/miniforge3/envs/ms-swift/lib/python3.11/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
[rank2]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 427, in wrapper
[rank2]: return func(self, *args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/trainers/rlhf_trainer/gkd_trainer.py", line 330, in training_step
[rank2]: loss = HFSFTTrainer.training_step(self, model, inputs, num_items_in_batch)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/miniforge3/envs/ms-swift/lib/python3.11/site-packages/trl/trainer/sft_trainer.py", line 914, in training_step
[rank2]: return super().training_step(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/miniforge3/envs/ms-swift/lib/python3.11/site-packages/transformers/trainer.py", line 3749, in training_step
[rank2]: loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 427, in wrapper
[rank2]: return func(self, *args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/trainers/rlhf_trainer/gkd_trainer.py", line 233, in compute_loss
[rank2]: loss = self.generalized_jsd_loss(
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/home/dugis/miniforge3/envs/ms-swift/lib/python3.11/site-packages/trl/trainer/gkd_trainer.py", line 191, in generalized_jsd_loss
[rank2]: mixture_log_probs = torch.logsumexp(
[rank2]: ^^^^^^^^^^^^^^^^
[rank2]: RuntimeError: CUDA driver error: invalid argument
[rank1]: Traceback (most recent call last):
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/cli/rlhf.py", line 7, in
[rank1]: rlhf_main()
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank1]: return SwiftRLHF(args).main()
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/llm/base.py", line 49, in main
[rank1]: result = self.run()
[rank1]: ^^^^^^^^^^
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank1]: return func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/llm/train/sft.py", line 196, in run
[rank1]: return self.train(trainer)
[rank1]: ^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/llm/train/sft.py", line 244, in train
[rank1]: trainer.train(trainer.args.resume_from_checkpoint)
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/trainers/mixin.py", line 816, in train
[rank1]: res = super().train(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/miniforge3/envs/ms-swift/lib/python3.11/site-packages/transformers/trainer.py", line 2206, in train
[rank1]: return inner_training_loop(
[rank1]: ^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/miniforge3/envs/ms-swift/lib/python3.11/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
[rank1]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 427, in wrapper
[rank1]: return func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/trainers/rlhf_trainer/gkd_trainer.py", line 330, in training_step
[rank1]: loss = HFSFTTrainer.training_step(self, model, inputs, num_items_in_batch)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/miniforge3/envs/ms-swift/lib/python3.11/site-packages/trl/trainer/sft_trainer.py", line 914, in training_step
[rank1]: return super().training_step(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/miniforge3/envs/ms-swift/lib/python3.11/site-packages/transformers/trainer.py", line 3749, in training_step
[rank1]: loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 427, in wrapper
[rank1]: return func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/shihaofeng/llm/ms-swift/swift/trainers/rlhf_trainer/gkd_trainer.py", line 233, in compute_loss
[rank1]: loss = self.generalized_jsd_loss(
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/dugis/miniforge3/envs/ms-swift/lib/python3.11/site-packages/trl/trainer/gkd_trainer.py", line 202, in generalized_jsd_loss
[rank1]: jsd = beta * kl_teacher + (1 - beta) * kl_student
[rank1]: ~~~~~~~~~~~^~~~~~~~~~~~
[rank1]: RuntimeError: CUDA driver error: invalid argument
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at swift/trainers/rlhf_trainer/gkd_trainer.py:233 and compare its generalized_jsd_loss call with the failing TRL gkd_trainer.py lines 191-202; reproduce through swift/cli/rlhf.py. Done means isolating a reproducible cause for the CUDA driver error and validating a fix in the GKD training path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100