facebookresearch / facebookresearch/dlrm

Criteo 1TB dataset is corrupted

Open
#413 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4.1k
Forks
859
PR merge metrics
No merged PRs in 30d

Description

The https://huggingface.co/datasets/criteo/CriteoClickLogs is missing two days of data and are truncated.
Should be (full):

- Compressed: ~15 GB per day × 24 days ≈ ~360 GB total
- Uncompressed: ~47 GB per day × 24 days ≈ ~1.1 TB total
- Rows: ~180 M per day × 24 days ≈ ~4.4 B total samples

Is now (HuggingFace):

- Compressed: ~1.6 GB per day × 22 days (day_0 and day_1 deleted) ≈ ~35 GB total
- Uncompressed (after handling truncation): ~5 GB per day × 22 days ≈ ~110 GB total
- Rows: ~20 M per day × 22 days ≈ ~440 M total samples

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.