alibaba / alibaba/bigcomputing

训练结果Test AUC过高的问题

Open
#1 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
33
Forks
12
PR merge metrics
No merged PRs in 30d

Description

1. 一般在ctr预测任务中,公开数据集auc很少能达到0.9以上,特别是这种真实的广告数据集,之前论文(如DIEN,DSIN等)一般为0.6-0.7之间。但是该数据集跑出来的结果甚至高于0.95,请问这是什么原因呢?
2. 训练数据集有0-15共16个npz文件,源码中作者只使用了前15个文件,我测试了使用全部16个文件后发现一个很奇怪的点,当开始训练最后一个文件时,test auc会有十分显著的提升,DIN模型可以从0.92提升到0.95。但是train_auc确没有什么提高,这是由于数据集划分时用了测试数据吗?

Contributor guide

No contributing guide indexed for this repository

Research direction

Compare the training behavior when using files 0–14 versus all 16 npz files, starting with the dataset split and the training loop that reports train and test AUC. Check whether the final file overlaps the test data or otherwise changes evaluation, and consider the investigation complete only when the AUC difference and the reported 0.95 result have an evidence-based explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.