alibaba / alibaba/EasyNLP

Spanproto recall为1.0的情况

Open
#248 4 comments 5 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.2k
Forks
257
PR merge metrics
No merged PRs in 30d

Description

您好!

在运行您提供的脚本后,发现recall一直为1.0的情况,不知是否是下面这行代码的原因(这里将真实标签也拼进去,后面评测的时候泄露了)https://github.com/wjn1996/SpanProto/blob/f1e0acb8672f0bfcbb7c827c48b06b3e8ccb295a/models/span_proto.py#L588

此外,在将此处改成`query_all_spans = query_predict_spans`后,得到的结果和论文中相差较大,不知哪边出了问题。
FEW-NERD 5way-1shot: inter—— span_f1:0.5826 class_f1:0.4618 intra——span_f1:0.4606 class_f1:0.3548

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at models/span_proto.py line 588 and run the provided evaluation script on FEW-NERD 5way-1shot. Check whether the true labels are included in query_all_spans, then compare the resulting inter and intra span_f1/class_f1 values with the reported paper results; done means the evaluation behavior and discrepancy are explained.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.