modelscope / modelscope/ms-swift

Unable to deploy fine-tuned Qwen3.5-35-A3B on vLLM

Open
#8,098 16 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Checklist / 检查清单
  • I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。
Bug Description / Bug 描述

After training a model using version 4.0.0dev from the main branch, the exported model has transformers_version set to 5.2.0. However, the current vLLM requires transformers<5, causing the following error during VLM deployment

[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26) Traceback (most recent call last):
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/bin/vllm", line 10, in <module>
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     sys.exit(main())
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)              ^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     args.dispatch_function(args)
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 112, in cmd
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     uvloop.run(run_server(args))
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     return __asyncio.run(
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)            ^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     return runner.run(main)
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)            ^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     return self._loop.run_until_complete(task)
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     return await main
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)            ^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 471, in run_server
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 490, in run_server_worker
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     async with build_async_engine_client(
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     return await anext(self.gen)
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)            ^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 96, in build_async_engine_client
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     async with build_async_engine_client_from_engine_args(
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     return await anext(self.gen)
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)            ^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 137, in build_async_engine_client_from_engine_args
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     async_llm = AsyncLLM.from_vllm_config(
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 223, in from_vllm_config
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     return cls(
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)            ^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 134, in __init__
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     self.renderer = renderer = renderer_from_config(self.vllm_config)
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/registry.py", line 88, in renderer_from_config
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     return RENDERER_REGISTRY.load_renderer(
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/registry.py", line 62, in load_renderer
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     return renderer_cls.from_config(config, tokenizer_kwargs)
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/renderers/hf.py", line 601, in from_config
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     cached_get_tokenizer(
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/registry.py", line 216, in get_tokenizer
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     tokenizer = tokenizer_cls_.from_pretrained(tokenizer_name, *args, **kwargs)
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/hf.py", line 110, in from_pretrained
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     raise e
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/hf.py", line 85, in from_pretrained
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     tokenizer = AutoTokenizer.from_pretrained(
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)   File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 1153, in from_pretrained
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26)     raise ValueError(
[2026-02-25 16:17:27] [/bin/sh]: (APIServer pid=26) ValueError: Tokenizer class TokenizersBackend does not exist or is not currently imported.
How to Reproduce / 如何复现

Swift 4.0.0dev
Transformers 5.2.0
vllm 0.16.0rc2.dev269+ga49ea5a58

Additional Information / 补充信息

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing deployment with Swift 4.0.0dev, Transformers 5.2.0, and vLLM 0.16.0rc2.dev269+ga49ea5a58. Trace the tokenizer-loading path shown in the vLLM stack trace, especially vllm/tokenizers/hf.py and transformers/models/auto/tokenization_auto.py. Done means the exported Qwen3.5-35-A3B model deploys without the TokenizersBackend error under a supported dependency combination.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.