modelscope / modelscope/ms-swift

qwen3.5 lora sft, vqa, error

Open
#8,160 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Checklist / 检查清单
  • I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。
Bug Description / Bug 描述
warmup_ratio=None,
warmup_steps=300.0,
weight_decay=0.1,
zero_hpz_partition_size=None,
)
[INFO:swift] Global seed set to 42
[INFO:swift] model_kwargs: {'device_map': None, 'dtype': torch.bfloat16}
[INFO:swift] attn_impl: flash_attn
[rank5]: Traceback (most recent call last):
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/cli/sft.py", line 20, in <module>
[rank5]:     sft_main()
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 354, in sft_main
[rank5]:     return SwiftSft(args).main()
[rank5]:            ^^^^^^^^^^^^^^
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 29, in __init__
[rank5]:     self._prepare_model_tokenizer()
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/ray/base.py", line 168, in wrapper
[rank5]:     return func(self, *args, **kwargs)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 51, in _prepare_model_tokenizer
[rank5]:     self.model, self.processor = args.get_model_processor(**kwargs)
[rank5]:                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/arguments/base_args/base_args.py", line 327, in get_model_processor
[rank5]:     return get_model_processor(**res)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 613, in get_model_processor
[rank5]:     return loader.load()
[rank5]:            ^^^^^^^^^^^^^
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 462, in load
[rank5]:     model, processor = self._get_model_processor(model_dir, config)
[rank5]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 450, in _get_model_processor
[rank5]:     processor = self.get_processor(model_dir, config)
[rank5]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/model/models/qwen.py", line 743, in get_processor
[rank5]:     processor = super().get_processor(model_dir, config)
[rank5]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 264, in get_processor
[rank5]:     return auto_tokenizer_cls.from_pretrained(model_dir, trust_remote_code=True)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/auto/processing_auto.py", line 402, in from_pretrained
[rank5]:     return processor_class.from_pretrained(
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1401, in from_pretrained
[rank5]:     return cls.from_args_and_dict(args, processor_dict, **instantiation_kwargs)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1168, in from_args_and_dict
[rank5]:     processor = cls(*args, **valid_kwargs)
[rank5]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/qwen3_vl/processing_qwen3_vl.py", line 60, in __init__
[rank5]:     super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 600, in __init__
[rank5]:     self.check_argument_for_proper_class(attribute_name, arg)
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in check_argument_for_proper_class
[rank5]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank5]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in <genexpr>
[rank5]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank5]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1557, in get_possibly_dynamic_module
[rank5]:     if hasattr(transformers_module, module_name):
[rank5]:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2044, in __getattr__
[rank5]:     module = self._get_module(self._class_to_module[name])
[rank5]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2238, in _get_module
[rank5]:     raise e
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2236, in _get_module
[rank5]:     return importlib.import_module("." + module_name, self.__name__)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/importlib/__init__.py", line 90, in import_module
[rank5]:     return _bootstrap._gcd_import(name[level:], package, level)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
[rank5]:   File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
[rank5]:   File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
[rank5]:   File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
[rank5]:   File "<frozen importlib._bootstrap_external>", line 994, in exec_module
[rank5]:   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
[rank5]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/tokenization_mistral_common.py", line 46, in <module>
[rank5]:     from mistral_common.tokens.tokenizers.utils import (
[rank5]: ImportError: cannot import name 'get_one_valid_tokenizer_file' from 'mistral_common.tokens.tokenizers.utils' (/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/utils.py). Did you mean: '_filter_valid_tokenizer_files'?
[rank7]: Traceback (most recent call last):
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/cli/sft.py", line 20, in <module>
[rank7]:     sft_main()
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 354, in sft_main
[rank7]:     return SwiftSft(args).main()
[rank7]:            ^^^^^^^^^^^^^^
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 29, in __init__
[rank7]:     self._prepare_model_tokenizer()
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/ray/base.py", line 168, in wrapper
[rank7]:     return func(self, *args, **kwargs)
[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 51, in _prepare_model_tokenizer
[rank7]:     self.model, self.processor = args.get_model_processor(**kwargs)
[rank7]:                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/arguments/base_args/base_args.py", line 327, in get_model_processor
[rank7]:     return get_model_processor(**res)
[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 613, in get_model_processor
[rank7]:     return loader.load()
[rank7]:            ^^^^^^^^^^^^^
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 462, in load
[rank7]:     model, processor = self._get_model_processor(model_dir, config)
[rank7]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 450, in _get_model_processor
[rank7]:     processor = self.get_processor(model_dir, config)
[rank7]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/model/models/qwen.py", line 743, in get_processor
[rank7]:     processor = super().get_processor(model_dir, config)
[rank7]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 264, in get_processor
[rank7]:     return auto_tokenizer_cls.from_pretrained(model_dir, trust_remote_code=True)
[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/auto/processing_auto.py", line 402, in from_pretrained
[rank7]:     return processor_class.from_pretrained(
[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1401, in from_pretrained
[rank7]:     return cls.from_args_and_dict(args, processor_dict, **instantiation_kwargs)
[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1168, in from_args_and_dict
[rank7]:     processor = cls(*args, **valid_kwargs)
[rank7]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/qwen3_vl/processing_qwen3_vl.py", line 60, in __init__
[rank7]:     super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 600, in __init__
[rank7]:     self.check_argument_for_proper_class(attribute_name, arg)
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in check_argument_for_proper_class
[rank7]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank7]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in <genexpr>
[rank7]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank7]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1557, in get_possibly_dynamic_module
[rank7]:     if hasattr(transformers_module, module_name):
[rank7]:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2044, in __getattr__
[rank7]:     module = self._get_module(self._class_to_module[name])
[rank7]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2238, in _get_module
[rank7]:     raise e
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2236, in _get_module
[rank7]:     return importlib.import_module("." + module_name, self.__name__)
[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/importlib/__init__.py", line 90, in import_module
[rank7]:     return _bootstrap._gcd_import(name[level:], package, level)
[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]:   File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
[rank7]:   File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
[rank7]:   File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
[rank7]:   File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
[rank7]:   File "<frozen importlib._bootstrap_external>", line 994, in exec_module
[rank7]:   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
[rank7]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/tokenization_mistral_common.py", line 46, in <module>
[rank7]:     from mistral_common.tokens.tokenizers.utils import (
[rank7]: ImportError: cannot import name 'get_one_valid_tokenizer_file' from 'mistral_common.tokens.tokenizers.utils' (/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/utils.py). Did you mean: '_filter_valid_tokenizer_files'?
[rank1]: Traceback (most recent call last):
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/cli/sft.py", line 20, in <module>
[rank1]:     sft_main()
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 354, in sft_main
[rank1]:     return SwiftSft(args).main()
[rank1]:            ^^^^^^^^^^^^^^
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 29, in __init__
[rank1]:     self._prepare_model_tokenizer()
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/ray/base.py", line 168, in wrapper
[rank1]:     return func(self, *args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 51, in _prepare_model_tokenizer
[rank1]:     self.model, self.processor = args.get_model_processor(**kwargs)
[rank1]:                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/arguments/base_args/base_args.py", line 327, in get_model_processor
[rank1]:     return get_model_processor(**res)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 613, in get_model_processor
[rank1]:     return loader.load()
[rank1]:            ^^^^^^^^^^^^^
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 462, in load
[rank1]:     model, processor = self._get_model_processor(model_dir, config)
[rank1]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 450, in _get_model_processor
[rank1]:     processor = self.get_processor(model_dir, config)
[rank1]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/model/models/qwen.py", line 743, in get_processor
[rank1]:     processor = super().get_processor(model_dir, config)
[rank1]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 264, in get_processor
[rank1]:     return auto_tokenizer_cls.from_pretrained(model_dir, trust_remote_code=True)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/auto/processing_auto.py", line 402, in from_pretrained
[rank1]:     return processor_class.from_pretrained(
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1401, in from_pretrained
[rank1]:     return cls.from_args_and_dict(args, processor_dict, **instantiation_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1168, in from_args_and_dict
[rank1]:     processor = cls(*args, **valid_kwargs)
[rank1]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/qwen3_vl/processing_qwen3_vl.py", line 60, in __init__
[rank1]:     super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 600, in __init__
[rank1]:     self.check_argument_for_proper_class(attribute_name, arg)
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in check_argument_for_proper_class
[rank1]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in <genexpr>
[rank1]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank1]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1557, in get_possibly_dynamic_module
[rank1]:     if hasattr(transformers_module, module_name):
[rank1]:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2044, in __getattr__
[rank1]:     module = self._get_module(self._class_to_module[name])
[rank1]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2238, in _get_module
[rank1]:     raise e
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2236, in _get_module
[rank1]:     return importlib.import_module("." + module_name, self.__name__)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/importlib/__init__.py", line 90, in import_module
[rank1]:     return _bootstrap._gcd_import(name[level:], package, level)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
[rank1]:   File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
[rank1]:   File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
[rank1]:   File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
[rank1]:   File "<frozen importlib._bootstrap_external>", line 994, in exec_module
[rank1]:   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
[rank1]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/tokenization_mistral_common.py", line 46, in <module>
[rank1]:     from mistral_common.tokens.tokenizers.utils import (
[rank1]: ImportError: cannot import name 'get_one_valid_tokenizer_file' from 'mistral_common.tokens.tokenizers.utils' (/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/utils.py). Did you mean: '_filter_valid_tokenizer_files'?
[rank3]: Traceback (most recent call last):
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/cli/sft.py", line 20, in <module>
[rank3]:     sft_main()
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 354, in sft_main
[rank3]:     return SwiftSft(args).main()
[rank3]:            ^^^^^^^^^^^^^^
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 29, in __init__
[rank3]:     self._prepare_model_tokenizer()
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/ray/base.py", line 168, in wrapper
[rank3]:     return func(self, *args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 51, in _prepare_model_tokenizer
[rank3]:     self.model, self.processor = args.get_model_processor(**kwargs)
[rank3]:                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/arguments/base_args/base_args.py", line 327, in get_model_processor
[rank3]:     return get_model_processor(**res)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 613, in get_model_processor
[rank3]:     return loader.load()
[rank3]:            ^^^^^^^^^^^^^
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 462, in load
[rank3]:     model, processor = self._get_model_processor(model_dir, config)
[rank3]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 450, in _get_model_processor
[rank3]:     processor = self.get_processor(model_dir, config)
[rank3]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/model/models/qwen.py", line 743, in get_processor
[rank3]:     processor = super().get_processor(model_dir, config)
[rank3]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 264, in get_processor
[rank3]:     return auto_tokenizer_cls.from_pretrained(model_dir, trust_remote_code=True)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/auto/processing_auto.py", line 402, in from_pretrained
[rank3]:     return processor_class.from_pretrained(
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1401, in from_pretrained
[rank3]:     return cls.from_args_and_dict(args, processor_dict, **instantiation_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1168, in from_args_and_dict
[rank3]:     processor = cls(*args, **valid_kwargs)
[rank3]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/qwen3_vl/processing_qwen3_vl.py", line 60, in __init__
[rank3]:     super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 600, in __init__
[rank3]:     self.check_argument_for_proper_class(attribute_name, arg)
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in check_argument_for_proper_class
[rank3]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank3]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in <genexpr>
[rank3]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank3]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1557, in get_possibly_dynamic_module
[rank3]:     if hasattr(transformers_module, module_name):
[rank3]:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2044, in __getattr__
[rank3]:     module = self._get_module(self._class_to_module[name])
[rank3]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2238, in _get_module
[rank3]:     raise e
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2236, in _get_module
[rank3]:     return importlib.import_module("." + module_name, self.__name__)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/importlib/__init__.py", line 90, in import_module
[rank3]:     return _bootstrap._gcd_import(name[level:], package, level)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
[rank3]:   File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
[rank3]:   File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
[rank3]:   File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
[rank3]:   File "<frozen importlib._bootstrap_external>", line 994, in exec_module
[rank3]:   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
[rank3]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/tokenization_mistral_common.py", line 46, in <module>
[rank3]:     from mistral_common.tokens.tokenizers.utils import (
[rank3]: ImportError: cannot import name 'get_one_valid_tokenizer_file' from 'mistral_common.tokens.tokenizers.utils' (/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/utils.py). Did you mean: '_filter_valid_tokenizer_files'?
[rank0]: Traceback (most recent call last):
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/cli/sft.py", line 20, in <module>
[rank0]:     sft_main()
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 354, in sft_main
[rank0]:     return SwiftSft(args).main()
[rank0]:            ^^^^^^^^^^^^^^
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 29, in __init__
[rank0]:     self._prepare_model_tokenizer()
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/ray/base.py", line 168, in wrapper
[rank0]:     return func(self, *args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 51, in _prepare_model_tokenizer
[rank0]:     self.model, self.processor = args.get_model_processor(**kwargs)
[rank0]:                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/arguments/base_args/base_args.py", line 327, in get_model_processor
[rank0]:     return get_model_processor(**res)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 613, in get_model_processor
[rank0]:     return loader.load()
[rank0]:            ^^^^^^^^^^^^^
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 462, in load
[rank0]:     model, processor = self._get_model_processor(model_dir, config)
[rank0]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 450, in _get_model_processor
[rank0]:     processor = self.get_processor(model_dir, config)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/model/models/qwen.py", line 743, in get_processor
[rank0]:     processor = super().get_processor(model_dir, config)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 264, in get_processor
[rank0]:     return auto_tokenizer_cls.from_pretrained(model_dir, trust_remote_code=True)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/auto/processing_auto.py", line 402, in from_pretrained
[rank0]:     return processor_class.from_pretrained(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1401, in from_pretrained
[rank0]:     return cls.from_args_and_dict(args, processor_dict, **instantiation_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1168, in from_args_and_dict
[rank0]:     processor = cls(*args, **valid_kwargs)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/qwen3_vl/processing_qwen3_vl.py", line 60, in __init__
[rank0]:     super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 600, in __init__
[rank0]:     self.check_argument_for_proper_class(attribute_name, arg)
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in check_argument_for_proper_class
[rank0]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in <genexpr>
[rank0]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank0]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1557, in get_possibly_dynamic_module
[rank0]:     if hasattr(transformers_module, module_name):
[rank0]:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2044, in __getattr__
[rank0]:     module = self._get_module(self._class_to_module[name])
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2238, in _get_module
[rank0]:     raise e
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2236, in _get_module
[rank0]:     return importlib.import_module("." + module_name, self.__name__)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/importlib/__init__.py", line 90, in import_module
[rank0]:     return _bootstrap._gcd_import(name[level:], package, level)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
[rank0]:   File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
[rank0]:   File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
[rank0]:   File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
[rank0]:   File "<frozen importlib._bootstrap_external>", line 994, in exec_module
[rank0]:   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
[rank0]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/tokenization_mistral_common.py", line 46, in <module>
[rank0]:     from mistral_common.tokens.tokenizers.utils import (
[rank0]: ImportError: cannot import name 'get_one_valid_tokenizer_file' from 'mistral_common.tokens.tokenizers.utils' (/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/utils.py). Did you mean: '_filter_valid_tokenizer_files'?
[rank4]: Traceback (most recent call last):
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/cli/sft.py", line 20, in <module>
[rank4]:     sft_main()
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 354, in sft_main
[rank4]:     return SwiftSft(args).main()
[rank4]:            ^^^^^^^^^^^^^^
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 29, in __init__
[rank4]:     self._prepare_model_tokenizer()
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/ray/base.py", line 168, in wrapper
[rank4]:     return func(self, *args, **kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 51, in _prepare_model_tokenizer
[rank4]:     self.model, self.processor = args.get_model_processor(**kwargs)
[rank4]:                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/arguments/base_args/base_args.py", line 327, in get_model_processor
[rank4]:     return get_model_processor(**res)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 613, in get_model_processor
[rank4]:     return loader.load()
[rank4]:            ^^^^^^^^^^^^^
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 462, in load
[rank4]:     model, processor = self._get_model_processor(model_dir, config)
[rank4]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 450, in _get_model_processor
[rank4]:     processor = self.get_processor(model_dir, config)
[rank4]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/model/models/qwen.py", line 743, in get_processor
[rank4]:     processor = super().get_processor(model_dir, config)
[rank4]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 264, in get_processor
[rank4]:     return auto_tokenizer_cls.from_pretrained(model_dir, trust_remote_code=True)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/auto/processing_auto.py", line 402, in from_pretrained
[rank4]:     return processor_class.from_pretrained(
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1401, in from_pretrained
[rank4]:     return cls.from_args_and_dict(args, processor_dict, **instantiation_kwargs)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1168, in from_args_and_dict
[rank4]:     processor = cls(*args, **valid_kwargs)
[rank4]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/qwen3_vl/processing_qwen3_vl.py", line 60, in __init__
[rank4]:     super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 600, in __init__
[rank4]:     self.check_argument_for_proper_class(attribute_name, arg)
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in check_argument_for_proper_class
[rank4]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank4]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in <genexpr>
[rank4]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank4]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1557, in get_possibly_dynamic_module
[rank4]:     if hasattr(transformers_module, module_name):
[rank4]:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2044, in __getattr__
[rank4]:     module = self._get_module(self._class_to_module[name])
[rank4]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2238, in _get_module
[rank4]:     raise e
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2236, in _get_module
[rank4]:     return importlib.import_module("." + module_name, self.__name__)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/importlib/__init__.py", line 90, in import_module
[rank4]:     return _bootstrap._gcd_import(name[level:], package, level)
[rank4]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]:   File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
[rank4]:   File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
[rank4]:   File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
[rank4]:   File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
[rank4]:   File "<frozen importlib._bootstrap_external>", line 994, in exec_module
[rank4]:   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
[rank4]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/tokenization_mistral_common.py", line 46, in <module>
[rank4]:     from mistral_common.tokens.tokenizers.utils import (
[rank4]: ImportError: cannot import name 'get_one_valid_tokenizer_file' from 'mistral_common.tokens.tokenizers.utils' (/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/utils.py). Did you mean: '_filter_valid_tokenizer_files'?
[rank6]: Traceback (most recent call last):
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/cli/sft.py", line 20, in <module>
[rank6]:     sft_main()
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 354, in sft_main
[rank6]:     return SwiftSft(args).main()
[rank6]:            ^^^^^^^^^^^^^^
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 29, in __init__
[rank6]:     self._prepare_model_tokenizer()
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/ray/base.py", line 168, in wrapper
[rank6]:     return func(self, *args, **kwargs)
[rank6]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 51, in _prepare_model_tokenizer
[rank6]:     self.model, self.processor = args.get_model_processor(**kwargs)
[rank6]:                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/arguments/base_args/base_args.py", line 327, in get_model_processor
[rank6]:     return get_model_processor(**res)
[rank6]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 613, in get_model_processor
[rank6]:     return loader.load()
[rank6]:            ^^^^^^^^^^^^^
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 462, in load
[rank6]:     model, processor = self._get_model_processor(model_dir, config)
[rank6]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 450, in _get_model_processor
[rank6]:     processor = self.get_processor(model_dir, config)
[rank6]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/model/models/qwen.py", line 743, in get_processor
[rank6]:     processor = super().get_processor(model_dir, config)
[rank6]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 264, in get_processor
[rank6]:     return auto_tokenizer_cls.from_pretrained(model_dir, trust_remote_code=True)
[rank6]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/auto/processing_auto.py", line 402, in from_pretrained
[rank6]:     return processor_class.from_pretrained(
[rank6]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1401, in from_pretrained
[rank6]:     return cls.from_args_and_dict(args, processor_dict, **instantiation_kwargs)
[rank6]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1168, in from_args_and_dict
[rank6]:     processor = cls(*args, **valid_kwargs)
[rank6]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/qwen3_vl/processing_qwen3_vl.py", line 60, in __init__
[rank6]:     super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 600, in __init__
[rank6]:     self.check_argument_for_proper_class(attribute_name, arg)
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in check_argument_for_proper_class
[rank6]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank6]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in <genexpr>
[rank6]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank6]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1557, in get_possibly_dynamic_module
[rank6]:     if hasattr(transformers_module, module_name):
[rank6]:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2044, in __getattr__
[rank6]:     module = self._get_module(self._class_to_module[name])
[rank6]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2238, in _get_module
[rank6]:     raise e
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2236, in _get_module
[rank6]:     return importlib.import_module("." + module_name, self.__name__)
[rank6]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/importlib/__init__.py", line 90, in import_module
[rank6]:     return _bootstrap._gcd_import(name[level:], package, level)
[rank6]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
[rank6]:   File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
[rank6]:   File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
[rank6]:   File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
[rank6]:   File "<frozen importlib._bootstrap_external>", line 994, in exec_module
[rank6]:   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
[rank6]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/tokenization_mistral_common.py", line 46, in <module>
[rank6]:     from mistral_common.tokens.tokenizers.utils import (
[rank6]: ImportError: cannot import name 'get_one_valid_tokenizer_file' from 'mistral_common.tokens.tokenizers.utils' (/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/utils.py). Did you mean: '_filter_valid_tokenizer_files'?
[rank2]: Traceback (most recent call last):
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/cli/sft.py", line 20, in <module>
[rank2]:     sft_main()
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 354, in sft_main
[rank2]:     return SwiftSft(args).main()
[rank2]:            ^^^^^^^^^^^^^^
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 29, in __init__
[rank2]:     self._prepare_model_tokenizer()
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/ray/base.py", line 168, in wrapper
[rank2]:     return func(self, *args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/pipelines/train/sft.py", line 51, in _prepare_model_tokenizer
[rank2]:     self.model, self.processor = args.get_model_processor(**kwargs)
[rank2]:                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/arguments/base_args/base_args.py", line 327, in get_model_processor
[rank2]:     return get_model_processor(**res)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 613, in get_model_processor
[rank2]:     return loader.load()
[rank2]:            ^^^^^^^^^^^^^
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 462, in load
[rank2]:     model, processor = self._get_model_processor(model_dir, config)
[rank2]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 450, in _get_model_processor
[rank2]:     processor = self.get_processor(model_dir, config)
[rank2]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/model/models/qwen.py", line 743, in get_processor
[rank2]:     processor = super().get_processor(model_dir, config)
[rank2]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/appdata/zhangkailin/ms-swift/swift/model/register.py", line 264, in get_processor
[rank2]:     return auto_tokenizer_cls.from_pretrained(model_dir, trust_remote_code=True)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/auto/processing_auto.py", line 402, in from_pretrained
[rank2]:     return processor_class.from_pretrained(
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1401, in from_pretrained
[rank2]:     return cls.from_args_and_dict(args, processor_dict, **instantiation_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1168, in from_args_and_dict
[rank2]:     processor = cls(*args, **valid_kwargs)
[rank2]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/models/qwen3_vl/processing_qwen3_vl.py", line 60, in __init__
[rank2]:     super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 600, in __init__
[rank2]:     self.check_argument_for_proper_class(attribute_name, arg)
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in check_argument_for_proper_class
[rank2]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank2]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 678, in <genexpr>
[rank2]:     proper_class = tuple(self.get_possibly_dynamic_module(n) for n in class_name if n is not None)
[rank2]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/processing_utils.py", line 1557, in get_possibly_dynamic_module
[rank2]:     if hasattr(transformers_module, module_name):
[rank2]:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2044, in __getattr__
[rank2]:     module = self._get_module(self._class_to_module[name])
[rank2]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2238, in _get_module
[rank2]:     raise e
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2236, in _get_module
[rank2]:     return importlib.import_module("." + module_name, self.__name__)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/importlib/__init__.py", line 90, in import_module
[rank2]:     return _bootstrap._gcd_import(name[level:], package, level)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
[rank2]:   File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
[rank2]:   File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
[rank2]:   File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
[rank2]:   File "<frozen importlib._bootstrap_external>", line 994, in exec_module
[rank2]:   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
[rank2]:   File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/transformers/tokenization_mistral_common.py", line 46, in <module>
[rank2]:     from mistral_common.tokens.tokenizers.utils import (
[rank2]: ImportError: cannot import name 'get_one_valid_tokenizer_file' from 'mistral_common.tokens.tokenizers.utils' (/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/utils.py). Did you mean: '_filter_valid_tokenizer_files'?
[rank0]:[W302 16:34:13.459249273 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
W0302 16:34:14.899000 2314934 /appdata/zhangkailin/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 2315010 closing signal SIGTERM
W0302 16:34:14.900000 2314934 /appdata/zhangkailin/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 2315011 closing signal SIGTERM
W0302 16:34:14.901000 2314934 /appdata/zhangkailin/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 2315012 closing signal SIGTERM
W0302 16:34:14.901000 2314934 /appdata/zhangkailin/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 2315014 closing signal SIGTERM
W0302 16:34:14.902000 2314934 /appdata/zhangkailin/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 2315015 closing signal SIGTERM
W0302 16:34:14.902000 2314934 /appdata/zhangkailin/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 2315016 closing signal SIGTERM
W0302 16:34:14.903000 2314934 /appdata/zhangkailin/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 2315017 closing signal SIGTERM
E0302 16:34:15.118000 2314934 /appdata/zhangkailin/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 3 (pid: 2315013) of binary: /home/zhangkailin/.conda/envs/swift/bin/python3.12
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/torch/distributed/run.py", line 905, in <module>
    main()
  File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 143, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zhangkailin/.conda/envs/swift/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 277, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/appdata/zhangkailin/ms-swift/swift/cli/sft.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-03-02_16:34:14
  host      : sit-gpu02
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 2315013)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
How to Reproduce / 如何复现

transformers-5.2.0,
ms-swift 源码安装

Additional Information / 补充信息

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at swift/cli/sft.py and follow the loading path through swift/pipelines/train/sft.py, swift/model/register.py, and swift/model/models/qwen.py. Reproduce the Qwen3-VL SFT startup failure and inspect the reported transformers and mistral_common import incompatibility. Done means the processor loads successfully without the get_one_valid_tokenizer_file ImportError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.