alibaba / alibaba/EasyRec

DropoutNet - Use official config & sample data but AUC and loss worsen with more training steps

Open
#513 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
383
PR merge metrics
No merged PRs in 30d

Description

## Description

I attempted to train the official DropoutNet model using the provided sample Taobao dataset and the sample configuration file. However, during training, I observed that the AUC decreased and the losses increased as the training steps progressed. Based on my understanding, the expected behavior is that the AUC should increase and the losses should decrease as training continues.

## Steps to reproduce

OS: Ubuntu 20.04
GPU: 1 NVIDIA RTX 3090
Python: 3.10.16
TensorFlow: 2.14.0 with CUDA

1. git clone the easyrec repo (commit SHA: https://github.com/alibaba/EasyRec/commit/4b0b1f5a2a990b253737ec532611fa4f9387d372)
2. install easyrec
3. download the sample taobao dataset:

```shell
wget http://easyrec.oss-cn-beijing.aliyuncs.com/data/git_oss_sample_data/data_test_tb_data_b1579db090d72b3b70b59ba3c7692701 -O tb_data.tar.gz
tar -zxf tb_data.tar.gz
```

4. run the training with the [sample dropoutnet config](https://github.com/alibaba/EasyRec/blob/4b0b1f5a2a990b253737ec532611fa4f9387d372/samples/model_config/dropoutnet_on_taobao.config) and sample dataset

```shell
python -m easy_rec.python.train_eval --pipeline_config_path samples/model_config/dropoutnet_on_taobao.config
```

## Actual training result

TensorBoard:

```shell
tensorboard --logdir experiments/dropoutnet_taobao_ckpt/eval_val
```

![image](https://github.com/user-attachments/assets/fdd1bff8-b926-4e6b-9614-82c9116f4612)

Initial AUC and loss:
![image](https://github.com/user-attachments/assets/e568553e-20c7-4d6a-a64d-46144ddd3a06)

Final AUC and loss:
![image](https://github.com/user-attachments/assets/393d0abf-1f90-43a5-ad04-1b2ddf5d545c)

## Expected behavior
- AUC should increase with more training steps.
- Losses should decrease with more training steps.

Could you please confirm if this is expected behavior or if there might be an issue with the sample configuration or dataset? If additional debugging information is needed, I am happy to provide more details.

Thank you!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running easy_rec.python.train_eval with samples/model_config/dropoutnet_on_taobao.config and the downloaded sample Taobao dataset. Inspect the TensorBoard logs under experiments/dropoutnet_taobao_ckpt/eval_val and compare AUC and loss across training steps. Done means determining whether the result is expected or providing evidence that the sample configuration or dataset is faulty.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.