Xtuner在自定义的数据集上运行报错
- Dominant language
- Python
- Stars
- 2k
- Forks
- 1.5k
- PR merge metrics
- No merged PRs in 30d
Description
环境配置都是按照教程来的。
然后自定义的数据集为:
{"conversation": [{"system": "你是一位旅游路线规划方向上,关键词提取的高手,你能够精确的获取到句子中的关键词,从而能够保证在网络上能够搜索到非常准确且良好的内容以供回复", "input": "我想要计划一个澳大利亚深度旅游,有哪些景点值得推荐?", "output": "澳大利亚, 深度旅游, 景点推荐"}]}
{"conversation": [{"system": "你是一位旅游路线规划方向上,关键词提取的高手,你能够精确的获取到句子中的关键词,从而能够保证在网络上能够搜索到非常准确且良好的内容以供回复", "input": "我准备去日本旅行,有什么好的行程规划建议?", "output": "日本, 旅行, 行程规划建议"}]}
{"conversation": [{"system": "你是一位旅游路线规划方向上,关键词提取的高手,你能够精确的获取到句子中的关键词,从而能够保证在网络上能够搜索到非常准确且良好的内容以供回复", "input": "我打算明年去巴黎旅游,需要提前订票吗?", "output": "巴黎, 旅游, 订票"}]}
总共有200多条。我是直接复制到MedQA上的。模型微调的时候成功运行起来了,其实可能由于数据太少了,运行的非常快,一两分钟就完成了。但是在转换为huggingface的时候报错了。
(xtuner0.1.9) root@intern-studio:~# export MKL_SERVICE_FORCE_INTEL=1
(xtuner0.1.9) root@intern-studio:~# xtuner convert pth_to_hf /root/ft-medqa/internlm_chat_7b_qlora_medqa2019_e3.py /root/ft-medqa/work_dirs/internlm_chat_7b_qlora_medqa2019_e3/epoch_3.pth /root/hf
[2024-01-10 14:41:22,702] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp-a34b3233.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
[2024-01-10 14:41:33,272] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
quantization_config convert to
Traceback (most recent call last):
File "/root/xtuner019/xtuner/xtuner/tools/model_converters/pth_to_hf.py", line 105, in
main()
File "/root/xtuner019/xtuner/xtuner/tools/model_converters/pth_to_hf.py", line 82, in main
model = BUILDER.build(cfg.model)
File "/root/.local/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/root/.local/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/root/xtuner019/xtuner/xtuner/model/sft.py", line 24, in __init__
self.llm = self._build_from_cfg_or_module(llm)
File "/root/xtuner019/xtuner/xtuner/model/sft.py", line 76, in _build_from_cfg_or_module
return BUILDER.build(cfg_or_mod)
File "/root/.local/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/root/.local/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/root/.local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 487, in from_pretrained
resolved_config_file = cached_file(
File "/root/.local/lib/python3.10/site-packages/transformers/utils/hub.py", line 429, in cached_file
resolved_file = hf_hub_download(
File "/root/.local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
validate_repo_id(arg_value)
File "/root/.local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 164, in validate_repo_id
raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: './internlm-chat-7b'.
现在不知道怎么来解决。可以指教一下吗?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.