modelscope / modelscope/ms-swift

自定义 metric 报错:UnpicklingError

Open
#6,658 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Describe the bug

SFT 训练评分模型(分数范围是 1, 1.5, 2, 2.5, 3),需要自定义 metric 计算 ACC 和 QWK。
数据格式(train valid test 格式相同):

{
    # LLM request messages
    "messages": [
        {"role": "system", "content": "..."},
        {"role": "user", "content": "..."},
        {"role": "assistant", "content": "..."}
    ],
    # Task-related info
    "UID": "...",
    "Score": "...",
    ...
}

复现步骤:

  1. 参考 示例代码 编写自定义 metric 如下:
from swift.plugin.metric import METRIC_MAPPING 

def compute_scoring_metrics(prediction) -> Dict[str, float]:
    preds, labels = prediction[0], prediction[1]
    new_preds, new_labels = [], []
    for i in range(preds.shape[0]):
        new_preds.append(Serializer.from_tensor(preds[i]))
        new_labels.append(Serializer.from_tensor(labels[i]))

    # ... 其他代码
    metric = get_scoring_metrics(new_preds, new_labels)
    return {"scoring": metric}

METRIC_MAPPING["scoring_metrics"] = (compute_scoring_metrics, None)
  1. swift sft 设置 --metric scoring_metrics 并运行

  2. !!报错:代码在 Serializer.from_tensor(preds[i]) 报错:_pickle.UnpicklingError: invalid load key '\x00'
    该错误来源于 swift.utils.torch_utils:Serializer.from_tensor(obj)pickle.loads(res[:buffer_size])

Your hardware and system info

  • H20 Ubuntu, CUDA 12
  • 使用两张卡微调

Additional context
出现上述报错后,尝试了一个 workaround:将 MODEL_ID_OR_PATH 存进环境变量,在 metric.py 中动态加载 tokenizer 将 token ids 还原成字符串。这个方法不会报错,但是出现另一个问题:模型训练时 eval 指标的值很高,acc 和 qwk 大于 0.9,但离线使用 vLLM 测试时这两个指标只有 0.2-0.3。推测是模型看到了 validation dataset 中 messages 的全部信息(不确定是不是),包括 assistant 字段信息,导致标签泄露?如果把 assistant 包含的分数改成一个不在有效分数区间的分数,eval 指标就会接近 0。

我的问题:

  1. 使用 repo 样例代码自定义 metric 为什么会报错?
  2. 自定义 metric 的 best practice 是什么?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with swift/plugin/metric.py and the referenced custom metric example, then inspect Serializer.from_tensor in swift/utils/torch_utils.py, especially its pickle.loads path. Reproduce the failure with the reported scoring metric and tensor inputs, and compare the training-time evaluation data with the offline vLLM inputs. Done means the cause of the unpickling error and the correct custom-metric usage are documented or fixed, with validation against the reported leakage behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.