internetarchive / internetarchive/openlibrary

Wikidata bulk data dump format is unwieldy

Open
#11,439 16 comments 0 reactions 0 assignees View on GitHub
Lead: @RayBB Module: Data dumps Priority: 3 Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 14h
Merged PRs (30d)
126

Description

### Problem

I'm not sure that providing dumps of someone else's data is advisable, but if you decide to keep doing it, it would be helpful to improve the format. The current dump is very difficult to use since it can't be processed with command line tools like `jq` the way all the rest of the OpenLibrary dumps are.

The current format is:
```

```
The funky double quoted JSON can't be used directly without unquoting it first. Aligning this format with that used by the rest of the dumps would make it a lot easier to handle.

~I'd recommend something like~
```

```
~putting the bulkiest field last~ (this should really be done in the table schema so that the dump matches the schema). ~You can probably also trim the time resolution. I'm not sure microsecond level precision is useful.~ (this is based on the psql datatype)

A one-liner like

```bash
gzcat ol_dump_wikidata... | cut -f 3 | jq -r . | less
```

Should produce parsed JSON that a developer can start exploring.

Also, the author dump contains a total 174961 copies of 166546 unique QIDs while the Wikidata dump contains 164090 lines. I'd expect all three of these numbers to be identical.

### Reproducing the bug

1. Go to ...
2. Do ...

* Expected behavior:
* Actual behavior:

### Context

- Browser (Chrome, Safari, Firefox, etc):
- OS (Windows, Mac, etc):
- Logged in (Y/N):
- Environment (prod, dev, local): prod

### Breakdown

#### Requirements Checklist
* [ ]

#### Related files

*

#### Stakeholders

*


#### Instructions for Contributors

- Please [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Research direction

Start by locating the code and schema that generate the Wikidata and author dumps, then compare their field order, quoting, and QID counts. Test the proposed pipeline with gzcat, cut, and jq against a generated dump. Done means the Wikidata output exposes raw parseable JSON in the agreed schema and the reported QID counts are reconciled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.