internetarchive / internetarchive/openlibrary
Add support for ISO 639-3 languages in edition cataloging
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Feature Request
## Problem / Opportunity
The global library community has (slowly) started it's move to extend cataloging language support to include all ISO 639-3 languages and I believe it's time for OpenLibrary to do the same. The [PCC guidelines adopted a few years ago](https://www.loc.gov/aba/pcc/scs/documents/ISO-639-3-guidelines.pdf) give good guidance on how to do this.
This was earlier mentioned in https://github.com/internetarchive/openlibrary/issues/8127#issuecomment-1652621459
## Proposal
- extend the language object schema to include identifiers for ISO 639-3 and ISO 639-2/B (ie the current MARC codes) identifiers
- populate those identifiers slots for all current languages
- look up languages using the identifiers rather than the `/languages/` to make the lookup code system independent
- ~identify any conflicts between ISO 639-2/B and ISO 639-2/T codes and identify what to do about them (ISO 639-3 is a superset of ISO 639-2/T, so will be covered by this analysis)~ EDIT: There is no overlap between ISO 639-2/B and ISO 639-3). Presumably these codes are reserved in perpetuity. There are 20 codes which are unique to MARC ie ISO 639-2/B.
- do the UX design for a language autocomplete that deals with the complexity of having thousands of languages (perhaps only add languages on demand as they're needed for cataloging?). This will have an important usability impact.
- import translated language labels for new languages from Wikidata
- update MARC importer to look for and utilize the code system tags on imported languages. When both MARC and ISO 639-3 codes are present the latter should be preferentially used. There will likely be cases where the MARC code is a macro language and the ISO 639-3 code is more specific.
- update the librarian cataloging instructions to provide guidance on how to select the right language
- ? do a bulk update of codes in existing records ? (not sure this is needed or even a good idea, but it's something that should be weighed) The 20 languages include some relatively high frequency ones, include French, German, and Dutch.
This will unblock requests like https://github.com/internetarchive/openlibrary/issues/7558
### Breakdown
Implementation Details (for maintainers)
#### Related files
Refer to [this map of common Endpoints](https://github.com/internetarchive/openlibrary/wiki/Endpoints):
*
#### Requirements Checklist
Checklist of requirements that need to be satisfied in order for this issue to be closed:
* [ ]
#### Stakeholders
* @hornc
* @mekarpeles
#### Instructions for Contributors
* **Before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) or pushing up changes to a PR, please first [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Assessment
This issue has not been assessed yet.