训练qwen2.5vl-7b时，使用evalscope在训练中评测速度非常慢

Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程，最好有截图)

训练脚本：

CUDA_VISIBLE_DEVICES=4,5,6,7
MAX_PIXELS=802816
MASTER_PORT=29610
NPROC_PER_NODE=4
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
swift sft
--model /data/liyc/Code/med_vl/lingshu_7b
--model_type qwen2_5_vl
--dataset /home/yuchong_li/M3D/CliVQA20k/dataset/mxqa_text_knowledge_qa.json
--attn_impl flash_attn
--train_type full
--torch_dtype bfloat16
--freeze_llm false
--freeze_vit true
--freeze_aligner true
--num_train_epochs 3
--per_device_train_batch_size 4
--learning_rate 1e-5
--gradient_accumulation_steps 8
--save_strategy epoch
--logging_steps 5
--max_length 16384
--output_dir /data/liyc/Output/lingshu_7b/mxqa_text_knowledge_qa/bs128_lr1e-5_freeze_vit_aligner
--warmup_ratio 0.05
--dataloader_num_workers 8
--dataset_num_proc 8
--deepspeed zero2
--eval_strategy steps
--eval_steps 100
--per_device_eval_batch_size 4
--eval_use_evalscope
--eval_dataset general_mcq
--eval_dataset_args '{"general_mcq": {"prompt_template": "Answer the following single-choice question and select the correct answer. Your entire response should be in this format: "Answer: [LETTER]" (without quotes), where [LETTER] is one of {letters}. Question: {question}\nOptions: {choices}", "local_path": "/home/yuchong_li/M3D/CliVQA20k/dataset/eval_dataset/mxqa-text_val.csv", "subset_list": ["mxqa-text"]}}'

eval：

gpu：

Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)

Package Version

absl-py 2.3.1
accelerate 1.9.0
addict 2.4.0
aiofiles 24.1.0
aiohappyeyeballs 2.6.1
aiohttp 3.12.14
aiosignal 1.4.0
aliyun-python-sdk-core 2.16.0
aliyun-python-sdk-kms 2.16.5
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 4.9.0
astor 0.8.1
astroid 4.0.2
asttokens 3.0.0
attrdict 2.0.1
attrs 25.3.0
av 15.0.0
beautifulsoup4 4.14.2
binpacking 1.5.2
bitsandbytes 0.46.1
black 25.12.0
blake3 1.0.5
Brotli 1.1.0
cachetools 6.1.0
cbor2 5.7.0
certifi 2025.7.14
cffi 1.17.1
charset-normalizer 3.4.2
click 8.2.1
cloudpickle 3.1.1
colorama 0.4.6
colorlog 6.10.1
compressed-tensors 0.10.2
contourpy 1.3.2
cpm-kernels 1.0.11
crcmod 1.7
cryptography 45.0.5
cupy-cuda12x 13.6.0
cycler 0.12.1
dacite 1.9.2
datasets 3.6.0
debugpy 1.8.18
decorator 5.2.1
decord 0.6.0
deepspeed 0.17.5
depyf 0.19.0
dill 0.3.8
diskcache 5.6.3
distro 1.9.0
dnspython 2.7.0
docstring_parser 0.17.0
dotenv 0.9.9
editdistance 0.8.1
einops 0.8.1
email_validator 2.2.0
et_xmlfile 2.0.0
evalscope 1.4.0
evaluate 0.4.6
eventlet 0.40.3
executing 2.2.1
fastapi 0.116.1
fastapi-cli 0.0.8
fastapi-cloud-cli 0.1.5
fastrlock 0.8.3
ffmpy 0.6.0
filelock 3.18.0
fire 0.7.1
flash_attn 2.8.0.post2
fonttools 4.59.0
frozenlist 1.7.0
fsspec 2024.12.0
func_timeout 4.3.5
future 1.0.0
fuzzywuzzy 0.18.0
gguf 0.17.1
gitdb 4.0.12
GitPython 3.1.45
google 3.0.0
google-auth 2.41.1
google-genai 1.41.0
gradio 5.38.0
gradio_client 1.11.0
greenlet 3.2.4
groovy 0.1.2
grpcio 1.73.1
h11 0.16.0
h5py 3.15.1
hf-xet 1.1.5
hjson 3.1.0
httpcore 1.0.9
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.34.4
human-eval 1.0.3
idna 3.10
ijson 3.4.0.post0
ImageIO 2.37.2
immutabledict 4.2.2
importlib_metadata 8.7.0
interegular 0.3.3
ipdb 0.13.13
ipython 9.5.0
ipython_pygments_lexers 1.1.1
isort 7.0.0
jedi 0.19.2
jieba 0.42.1
Jinja2 3.1.6
jiter 0.10.0
jmespath 0.10.0
joblib 1.5.1
json_repair 0.54.2
json5 0.12.1
jsonlines 4.0.0
jsonschema 4.25.1
jsonschema-specifications 2025.4.1
kiwisolver 1.4.8
kneed 0.8.5
langdetect 1.0.9
lark 1.2.2
latex2sympy2_extended 1.10.2
Levenshtein 0.27.3
librt 0.7.3
llguidance 0.7.30
llvmlite 0.44.0
lm-format-enforcer 0.10.12
lxml 6.0.1
Markdown 3.8.2
markdown-it-py 3.0.0
MarkupSafe 3.0.2
math-verify 0.8.0
mathruler 0.1.0
matplotlib 3.10.3
matplotlib-inline 0.1.7
mccabe 0.7.0
mdurl 0.1.2
mistral_common 1.8.3
mmengine-lite 0.10.7
modelscope 1.28.0
more-itertools 10.8.0
mpmath 1.3.0
ms-opencompass 0.1.6
ms_swift 3.11.1
ms-vlmeval 0.0.19
msgpack 1.1.1
msgspec 0.19.0
multidict 6.6.3
multiprocess 0.70.16
mypy 1.19.0
mypy_extensions 1.1.0
networkx 3.5
ninja 1.11.1.4
nltk 3.9.1
numba 0.61.2
numpy 1.26.4
nvidia-cublas-cu12 12.6.4.1
nvidia-cuda-cupti-cu12 12.6.80
nvidia-cuda-nvrtc-cu12 12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12 9.5.1.17
nvidia-cufft-cu12 11.3.0.4
nvidia-cufile-cu12 1.11.1.6
nvidia-curand-cu12 10.3.7.77
nvidia-cusolver-cu12 11.7.1.2
nvidia-cusparse-cu12 12.5.4.2
nvidia-cusparselt-cu12 0.6.3
nvidia-ml-py 12.575.51
nvidia-nccl-cu12 2.26.2
nvidia-nvjitlink-cu12 12.6.85
nvidia-nvtx-cu12 12.6.77
omegaconf 2.3.0
openai 1.90.0
openai-harmony 0.0.4
OpenCC 1.1.9
opencv-python 4.11.0.86
opencv-python-headless 4.12.0.88
openpyxl 3.1.5
orjson 3.11.0
oss2 2.19.1
outlines_core 0.2.10
overrides 7.7.0
packaging 25.0
pandas 2.3.1
parso 0.8.5
partial-json-parser 0.2.1.1.post6
PasteDeploy 3.1.0
pathspec 0.12.1
peft 0.15.2
pexpect 4.9.0
pillow 11.3.0
pip 25.1
platformdirs 4.3.8
portalocker 3.2.0
prettytable 3.17.0
prometheus_client 0.22.1
prometheus-fastapi-instrumentator 7.1.0
prompt_toolkit 3.0.52
propcache 0.3.2
protobuf 6.31.1
psutil 7.0.0
ptyprocess 0.7.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
pyarrow 21.0.0
pyasn1 0.6.1
pyasn1_modules 0.4.2
pybase64 1.4.2
pycountry 24.6.1
pycparser 2.22
pycryptodome 3.23.0
pydantic 2.11.7
pydantic_core 2.33.2
pydantic-extra-types 2.10.5
pydub 0.25.1
pyeclib 1.7.0
Pygments 2.19.2
pylatexenc 2.10
pylint 4.0.4
PyMuPDF 1.26.4
pyparsing 3.2.3
pypinyin 0.55.0
python-dateutil 2.9.0.post0
python-dotenv 1.1.1
python-json-logger 3.3.0
python-Levenshtein 0.27.3
python-multipart 0.0.20
pytokens 0.3.0
pytz 2025.2
PyYAML 6.0.2
pyzmq 27.0.1
qwen-vl-utils 0.0.11
rank-bm25 0.2.2
RapidFuzz 3.14.3
ray 2.48.0
referencing 0.36.2
regex 2024.11.6
requests 2.32.4
rich 14.0.0
rich-toolkit 0.15.0
rignore 0.6.4
rouge 1.0.1
rouge-chinese 1.0.3
rouge_score 0.1.2
rpds-py 0.27.0
rsa 4.9.1
ruff 0.12.4
sacrebleu 2.5.1
safehttpx 0.1.6
safetensors 0.5.3
scikit-learn 1.7.1
scipy 1.16.0
seaborn 0.13.2
semantic-version 2.10.0
sentence-transformers 5.2.0
sentencepiece 0.2.0
sentry-sdk 2.35.0
setproctitle 1.3.6
setuptools 78.1.1
shellingham 1.5.4
simplejson 3.20.1
six 1.17.0
smmap 5.0.2
sniffio 1.3.1
sortedcontainers 2.4.0
soundfile 0.13.1
soupsieve 2.8
soxr 0.5.0.post1
sse-starlette 3.0.3
stack-data 0.6.3
starlette 0.47.2
sty 1.0.6
swift 2.36.0
sympy 1.14.0
tabulate 0.9.0
tenacity 9.1.2
tensorboard 2.20.0
tensorboard-data-server 0.7.2
termcolor 3.2.0
threadpoolctl 3.6.0
tiktoken 0.9.0
timeout-decorator 0.5.0
timm 1.0.22
tokenizers 0.21.4
tomlkit 0.13.3
torch 2.7.1
torchaudio 2.7.1
torchvision 0.22.1
tqdm 4.67.1
traitlets 5.14.3
transformers 4.55.2
transformers-stream-generator 0.0.5
triton 3.3.1
trl 0.19.1
typer 0.16.0
typing_extensions 4.14.1
typing-inspection 0.4.1
tzdata 2025.2
urllib3 2.5.0
uvicorn 0.35.0
uvloop 0.21.0
validators 0.35.0
vllm 0.10.0
wandb 0.21.1
watchfiles 1.1.0
wcwidth 0.2.13
websockets 15.0.1
Werkzeug 3.1.3
wheel 0.45.1
word2number 1.1
xattr 1.2.0
xformers 0.0.31
xgrammar 0.1.21
xlsxwriter 3.2.9
xxhash 3.5.0
yapf 0.43.0
yarl 1.20.1
zhconv 1.4.3
zipp 3.23.0
zstandard 0.23.0

Additional context
Add any other context about the problem here(在这里补充其他信息)

感觉像是在用cpu做eval

Start by reproducing the swift sft command with --eval_strategy steps, --eval_steps 100, and --eval_use_evalscope, then inspect EvalScope evaluation behavior and the reported GPU utilization. Compare training and evaluation timings across the four GPUs and document or resolve the slowdown while preserving the intended dataset and evaluation settings.