mlcommons / mlcommons/inference
DLRM v2 Preprocessed Multihot Criteo day_23 Dataset Accuracy Drop
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
From the step at https://github.com/mlcommons/training/blob/3a6a379305e6ef0a1c34461c823e9c99d52c1021/recommendation_v2/torchrec_dlrm/scripts/process_Criteo_1TB_Click_Logs_dataset.sh#L42, there was a difference with the generated day_23_sparse.py using torchrec==0.3.2 resulting in an roc_auc of 61.64%
expected row_0 array([[ 10540786, 197, 34, ..., 34, 2, 3]
resulting row_0 array([[ 449831406, 456128031, 780871217, ..., 374479166, 809724924, -1218975401],
The subsequent day_23_sparse_multi_hot.npz is therefore also incorrect
day_23_dense.npy and day_23_labels.npy have the correct md5 as here https://github.com/mlcommons/training/blob/3a6a379305e6ef0a1c34461c823e9c99d52c1021/recommendation_v2/torchrec_dlrm/md5sums_preprocessed_criteo_click_logs_dataset.txt#L70
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read recommendation_v2/torchrec_dlrm/scripts/process_Criteo_1TB_Click_Logs_dataset.sh at the linked step and inspect how day_23_sparse.py is generated with torchrec==0.3.2. Compare its row_0 values with the expected array, then trace the resulting day_23_sparse_multi_hot.npz; completion should explain or correct the sparse-data discrepancy while preserving the matching day_23_dense.npy and day_23_labels.npy checksums.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100