internetarchive / internetarchive/openlibrary

Community Batch Import Request: arabooks

Open
#9,726 27 comments 0 reactions 1 assignee Claimed by @momna-ahmad View on GitHub
Good First Issue Lead: @mekarpeles Module: Import Needs: Response Priority: 3 Type: Question
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

I have a repository of +12K Arabic books metadata: https://github.com/avidseeker/arabooks

If there is a bot to mass-upload them that would be a great addition to OpenLibrary. It currently lacks a lot of Arabic books coverage. (There isn't even an Arabic translation for OpenLibrary: https://github.com/internetarchive/openlibrary/pull/9673)

Thanks in advance.

---
Edit:
To complete this issue one would need to parse the TSV files found at https://github.com/avidseeker/arabooks and create JSONL files that look similar to this:

```json
{"identifiers": {"open_textbook_library": ["1581"]}, "source_records": ["open_textbook_library:1581"], "title": "Legal Fundamentals of Healthcare Law", "languages": ["eng"], "subjects": ["Medicine", "Law"], "publishers": ["University of West Florida Pressbooks"], "publish_date": "2024", "authors": [{"name": "Tiffany Jackman"}], "lc_classifications": ["RA440", "KF385.A4"]}
{"identifiers": {"open_textbook_library": ["1580"]}, "source_records": ["open_textbook_library:1580"], "title": "Introduction to Literature: Fairy Tales, Folk Tales, and How They Shape Us", "languages": ["eng"], "subjects": ["Humanities", "Literature, Rhetoric, and Poetry"], "publishers": ["University of West Florida Pressbooks"], "publish_date": "2023", "authors": [{"name": "Judy Young"}], "lc_classifications": ["PE1408"]}
```

The minimum required fields are: `title`, `authors`, `publish_date`, `source_records`, and `publish_date`. The part of the `source_records` could come from the name of the source, and an identifier, such as for [loal-en.tsv](https://github.com/avidseeker/arabooks/blob/main/loal-en.tsv) from the Library of Arabic Literature, it might be `"source_records": ["loal:9781479834129"]` for the first item in the list.

Here are the publishers of the TSV files:

1. awu-dam.tsv: Arabi Writers Union
2. lisanarb.tsv: contains a `pub` entry
3. loal-en.tsv and loal-ar.tsv: Library of Arabic Literature
4. shamela.tsv: contains a `publisher` entry. Dates need to be merged from
shamela-dates.tsv matching same title entry.
5. waqfeya.tsv: set as `"publishers": ["????"]`, since publishers need to be
known on one by one basis.

Specifically, the values taken from the TSV and converted into JSONL would need to follow [this schema](https://github.com/internetarchive/openlibrary-client/blob/master/olclient/schemata/import.schema.json). A script to do this for one line would look similar to [this](https://github.com/internetarchive/openlibrary/blob/153cd28a604a1152f3a1755eec346dd4c35ac728/scripts/import_open_textbook_library.py#L30-L112), but would probably use Python's `csv` module to read the TSV file, and then call `json.dumps(line)` on each line, after the data is in format specified in the import schema, and then it would be written to a JSONL file.

The output JSONL file could be tested using the endpoint from #8122, though you'd probably want to test with only a few records at a time rather than the whole file.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.