internetarchive / internetarchive/openlibrary
Wikidata bulk data dump format is unwieldy
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 126
Description
### Problem
I'm not sure that providing dumps of someone else's data is advisable, but if you decide to keep doing it, it would be helpful to improve the format. The current dump is very difficult to use since it can't be processed with command line tools like `jq` the way all the rest of the OpenLibrary dumps are.
The current format is:
```
```
The funky double quoted JSON can't be used directly without unquoting it first. Aligning this format with that used by the rest of the dumps would make it a lot easier to handle.
~I'd recommend something like~
```
```
~putting the bulkiest field last~ (this should really be done in the table schema so that the dump matches the schema). ~You can probably also trim the time resolution. I'm not sure microsecond level precision is useful.~ (this is based on the psql datatype)
A one-liner like
```bash
gzcat ol_dump_wikidata... | cut -f 3 | jq -r . | less
```
Should produce parsed JSON that a developer can start exploring.
Also, the author dump contains a total 174961 copies of 166546 unique QIDs while the Wikidata dump contains 164090 lines. I'd expect all three of these numbers to be identical.
### Reproducing the bug
1. Go to ...
2. Do ...
* Expected behavior:
* Actual behavior:
### Context
- Browser (Chrome, Safari, Firefox, etc):
- OS (Windows, Mac, etc):
- Logged in (Y/N):
- Environment (prod, dev, local): prod
### Breakdown
#### Requirements Checklist
* [ ]
#### Related files
*
#### Stakeholders
*
#### Instructions for Contributors
- Please [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Research direction
Start by locating the code and schema that generate the Wikidata and author dumps, then compare their field order, quoting, and QID counts. Test the proposed pipeline with gzcat, cut, and jq against a generated dump. Done means the Wikidata output exposes raw parseable JSON in the agreed schema and the reported QID counts are reconciled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100