14% of SHA256 hashes not matching

Introduction

We downloaded the Datacomp 1B set.
For verification, we only kept an image if its SHA256 checksum of the bytes matches with the corresponding entry in the metadata you provide.

Problem Statement

Hundreds of millions of images were discarded due to hash mismatches.

Here's one example. Let's look at entry 21:
https://huggingface.co/datasets/mlfoundations/datacomp_1b/viewer/default/train?row=21

• UID: 38f76e4b1b4a77ca66a62b453da17912
• text: Cable Manager, Horizontal, Recessed Flat ...
• url: https://images.eanixter.com/viewex/PR108844V6.JPG
• sha256: 0e77fada7a972fc2fa2ad8991701968ea0953d03f799787c4d27f072bdfa4164

If you download the image and compute the hash, it will be this:

$ curl -s https://images.eanixter.com/viewex/PR108844V6.JPG | sha256sum
6f392bcd683005e33356a2c003fb8378c7ff445d0b54b3429c84bd112299d273

One might think the image was modified slightly (e.g. its header or some pixels). However, checking the web archive version from 2019 yields the same hash:

$ curl -s https://web.archive.org/web/20191127193532if_/https://images.eanixter.com/viewex/PR108844V6.JPG | sha256sum
6f392bcd683005e33356a2c003fb8378c7ff445d0b54b3429c84bd112299d273

(You can view the web archive capture here.)

Mitigation

We would like to better understand how the hashes were computed. It seems the code that was used for that is not published.
Potentially, we could build a workaround by computing the hashes in the same way you did.

Ultimately, we think fixing the hashes in the metadata will be the best solution.

Start by reproducing the entry-21 mismatch using the issue's image URL, metadata SHA256, curl command, and web archive URL. Investigate how the hashes were computed, since the issue states that the hashing code is not published; done means the computation discrepancy is explained and a path to correcting the metadata is established.