allenai / allenai/mmc4

Missing or broken images (due to stale URLs)

Offen
#14 8 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
954
Forks
37
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

> For raw images, the main readme has a raw image interest list. For copyright/legal reasons, I can't directly distribute images. Can you provide some statistics about what percentage of missing images you're finding? If a very high number are missing, I can do more thinking about potential solutions.

_Originally posted by @jmhessel in https://github.com/allenai/mmc4/issues/10#issuecomment-1550204615_

To answer your question:
I checked how many docs/samples there are in the originally published jsonl files, vs. how many intact docs we were able to extract.
For the full dataset (incl. faces), the percentage of missing samples is 16.8%, so quite high.
I counted the unique URLs in the original data and the unique URLs in our dataset which has been filtered for

- Samples that are missing one or more images (could not be downloaded)
- Samples that contain images that cannot be loaded, or decoded, or are missing part of the image

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.