Improve translation of ROR organization names

Problem

When we wrote the first ROR import, we did not handle organization names in multiple languages via django-modeltranslation because some organizations in the ROR data have multiple "labels" per language. In ROR's schema, 'label' is a preferred name, so theoretically it should be "display-ready".

To see the records with duplicates, you can use a command like this, and then view the JSON record linked from the ROR webpage. Interestingly, ROR does not display more than one label per language, despite the JSON data having multiples.

# src/utils/management/commands/profile_ror_data.py
import zipfile
import json

from django.conf import settings
from django.core.management.base import BaseCommand
from django.utils import timezone

from utils.models import RORImport
from core.models import Organization
from utils.logger import get_logger


logger = get_logger(__name__)


class Command(BaseCommand):
    """
    Fetches ROR data and generates Organization records.
    """

    help = "Fetches ROR data and generates Organization records."

    def add_arguments(self, parser):
        return super().add_arguments(parser)

    def handle(self, *args, **options):
        ror_import = RORImport.objects.create()
        ror_import.get_records()

        if ror_import.is_ongoing:
            if not ror_import.previous_import:
                ror_import.download_data()
            elif ror_import.previous_import.zip_path != ror_import.zip_path:
                ror_import.download_data()

        if not ror_import.is_ongoing:
            return

        with zipfile.ZipFile(ror_import.zip_path, mode='r') as zip_ref:
            for file_info in zip_ref.infolist():
                if file_info.filename.endswith('v2.json'):
                    json_string = zip_ref.read(file_info).decode(encoding="utf-8")
                    data = json.loads(json_string)

        duplicate_labels_by_lang = []
        for record in data:
            labels = {}
            status = record.get('status', '')
            if status != 'active':
                continue
            for name in record.get('names'):
                for name_type in name.get('types', ''):
                    if name_type != 'label':
                        continue
                    value = name.get('value', '')
                    lang = name.get('lang', '')
                    if not lang:
                        continue
                    if lang not in labels:
                        labels[lang] = value
                    else:
                        ror_id = record.get('id', '')
                        duplicate_labels_by_lang.append(ror_id)
        print(duplicate_labels_by_lang)

Sample data:

['https://ror.org/022d5qt08', 'https://ror.org/03bwf4y78', 'https://ror.org/05kzadn81', 'https://ror.org/00vxe3x87', 'https://ror.org/04d7es448', 'https://ror.org/04m3xd186', 'https://ror.org/00kays408', 'https://ror.org/01y4xm534', 'https://ror.org/05c2p1f98', 'https://ror.org/008hybe55', 'https://ror.org/00t4vnv68', 'https://ror.org/02hbrab76', 'https://ror.org/04hk86037', 'https://ror.org/04qdwp261', 'https://ror.org/05c9p1x46', 'https://ror.org/001wpa366', 'https://ror.org/05fz2yc38', 'https://ror.org/0398yj006', 'https://ror.org/01d012874', 'https://ror.org/02pb4g840', 'https://ror.org/01jj26143', 'https://ror.org/04r17y386', 'https://ror.org/03ehse820', 'https://ror.org/01rje3r53', 'https://ror.org/04vh1tq58', 'https://ror.org/02aj0wy64', 'https://ror.org/01bnjyq25', 'https://ror.org/00kftxm09', 'https://ror.org/04xmnzw38', 'https://ror.org/01strh679', 'https://ror.org/05eyymj94', 'https://ror.org/03t1jzs40', 'https://ror.org/01ttgmj63', 'https://ror.org/017fh2655', 'https://ror.org/00j707644', 'https://ror.org/0232mk144', 'https://ror.org/03vpkt967', 'https://ror.org/03by7ks57', 'https://ror.org/02p7gn625', 'https://ror.org/02m805769', 'https://ror.org/040hka494', 'https://ror.org/012xcvh54', 'https://ror.org/01601en76', 'https://ror.org/05bsers28', 'https://ror.org/02es2jq57', 'https://ror.org/02jqwca63', 'https://ror.org/04bwfkw08', 'https://ror.org/042qbkd11', 'https://ror.org/02mx39t82', 'https://ror.org/05xzvft44', 'https://ror.org/01mvsby80', 'https://ror.org/042f0eq65', 'https://ror.org/05rynnb36', 'https://ror.org/0163bsq78', 'https://ror.org/00ghzk478', 'https://ror.org/056m1n765', 'https://ror.org/00yssnc44', 'https://ror.org/038t94116', 'https://ror.org/021swwa08', 'https://ror.org/05g6deb76', 'https://ror.org/03r4w0b84', 'https://ror.org/0090j2029', 'https://ror.org/031h57r44', 'https://ror.org/054zgtc98', 'https://ror.org/00g6pb267', 'https://ror.org/00n855a93', 'https://ror.org/008a3c408', 'https://ror.org/00wtksj25', 'https://ror.org/04bjh1v02', 'https://ror.org/02h8ngt70', 'https://ror.org/01qek4j34', 'https://ror.org/003bfq559', 'https://ror.org/05cyjba52', 'https://ror.org/044jjd544', 'https://ror.org/0144xq035', 'https://ror.org/00hj48m64', 'https://ror.org/01k4bzs92', 'https://ror.org/03x43nn58', 'https://ror.org/034vsyd62', 'https://ror.org/05h22s698', 'https://ror.org/03cyqfb07', 'https://ror.org/007h7f935', 'https://ror.org/03f099r71', 'https://ror.org/057g08s23', 'https://ror.org/054v4yq51', 'https://ror.org/01r98zz61', 'https://ror.org/0573cnq15', 'https://ror.org/02vg7rh06', 'https://ror.org/05kkct931', 'https://ror.org/040hyy284', 'https://ror.org/047a24r47', 'https://ror.org/03jzm5a44', 'https://ror.org/00d7rqp94', 'https://ror.org/059hjnz30', 'https://ror.org/058aanc53', 'https://ror.org/03deqdj72', 'https://ror.org/03ye93e96', 'https://ror.org/00n7m6g17', 'https://ror.org/02bnw0f54', 'https://ror.org/0057ag334', 'https://ror.org/03ez88s78', 'https://ror.org/049ysg747', 'https://ror.org/0100d8y50', 'https://ror.org/00m2sk374', 'https://ror.org/01gm99y36', 'https://ror.org/04pad1p35', 'https://ror.org/05djcr692', 'https://ror.org/01rbdtr54', 'https://ror.org/01fqmwr23', 'https://ror.org/010pg3352', 'https://ror.org/006eggh59', 'https://ror.org/05th78b98', 'https://ror.org/012w6mj89', 'https://ror.org/055zdhj54', 'https://ror.org/02be22443', 'https://ror.org/055pgv213', 'https://ror.org/05vjvk624', 'https://ror.org/016fy5v41', 'https://ror.org/050d4yq46', 'https://ror.org/02j25ht90', 'https://ror.org/035c9qf67', 'https://ror.org/00zam0e96', 'https://ror.org/01kd5m353', 'https://ror.org/00dkye506', 'https://ror.org/048nn3q69', 'https://ror.org/02er3bg49', 'https://ror.org/04atwyt60', 'https://ror.org/03pephy40', 'https://ror.org/04s85wv17', 'https://ror.org/024a7ry57', 'https://ror.org/01cdn3r29', 'https://ror.org/02j0abw33', 'https://ror.org/00ppmht14', 'https://ror.org/03bke4038', 'https://ror.org/04spvzc70', 'https://ror.org/034hztf92', 'https://ror.org/02d0cnn24', 'https://ror.org/05x3pnd66', 'https://ror.org/0368f8s77', 'https://ror.org/0259gsa43', 'https://ror.org/0259bcz98', 'https://ror.org/0458ana96', 'https://ror.org/03vxb8t23', 'https://ror.org/004gk0w56', 'https://ror.org/025fme611', 'https://ror.org/02qx7s056', 'https://ror.org/022wwx668', 'https://ror.org/058mfk884', 'https://ror.org/02p8nt844', 'https://ror.org/03y1zyv86', 'https://ror.org/01wm9rt41', 'https://ror.org/01kwczx50', 'https://ror.org/03ae9x524', 'https://ror.org/001yxxw51', 'https://ror.org/04jfz0g97', 'https://ror.org/041621784', 'https://ror.org/01hzrxx48', 'https://ror.org/04ht9zz88', 'https://ror.org/03fh39287', 'https://ror.org/01tanx198', 'https://ror.org/00qgwdb66', 'https://ror.org/01855q607', 'https://ror.org/03kk24v11', 'https://ror.org/00xh26g31', 'https://ror.org/04sgttv52', 'https://ror.org/00we1pa83', 'https://ror.org/01pg7ze09', 'https://ror.org/00jdhb158', 'https://ror.org/053n55597', 'https://ror.org/018t8yw14', 'https://ror.org/0546v0856', 'https://ror.org/02zxpkz02', 'https://ror.org/03cmq3458', 'https://ror.org/03n0qtw28', 'https://ror.org/05eyxqf74', 'https://ror.org/0522bga53', 'https://ror.org/005cx3d93', 'https://ror.org/02c8msm58', 'https://ror.org/033d3q980', 'https://ror.org/04q01pf08', 'https://ror.org/003cma469', 'https://ror.org/02q0t9634', 'https://ror.org/01pfpr664', 'https://ror.org/05ynf4333', 'https://ror.org/03abgds10', 'https://ror.org/04qchsx62', 'https://ror.org/03n8qgc13', 'https://ror.org/00ztcbk21', 'https://ror.org/00z4gpq27', 'https://ror.org/030c55a26', 'https://ror.org/034k6e791', 'https://ror.org/04jjswc10', 'https://ror.org/02hpe9z95', 'https://ror.org/045nsn047', 'https://ror.org/01drtms03', 'https://ror.org/03p5hcj07', 'https://ror.org/043nq9007', 'https://ror.org/03agmyp46', 'https://ror.org/05rm1p636', 'https://ror.org/011bkgz47', 'https://ror.org/02704zf98', 'https://ror.org/0232zwb84', 'https://ror.org/02p3csg43', 'https://ror.org/01wvk2989', 'https://ror.org/0306thw41', 'https://ror.org/049zex268', 'https://ror.org/01ttwsg70', 'https://ror.org/05g452t43', 'https://ror.org/046xvb515', 'https://ror.org/05ybghm69', 'https://ror.org/056kpdx27', 'https://ror.org/02ebbk435', 'https://ror.org/03r6kns77', 'https://ror.org/04wnzzd87', 'https://ror.org/0014n6t23', 'https://ror.org/00zq17y52', 'https://ror.org/01d16dr57', 'https://ror.org/05mj8hk83', 'https://ror.org/02xq7zd76', 'https://ror.org/00mt9mq23', 'https://ror.org/008gfyt86', 'https://ror.org/00zb6nk96', 'https://ror.org/04kthcs16', 'https://ror.org/05j2csz34', 'https://ror.org/05j242h88', 'https://ror.org/04rhps755', 'https://ror.org/05ce10k61', 'https://ror.org/00kr3hh47', 'https://ror.org/01kk1vy78', 'https://ror.org/016018d94', 'https://ror.org/010q0m202', 'https://ror.org/02v7y7w12', 'https://ror.org/04ykg4p88', 'https://ror.org/05jdrrw50', 'https://ror.org/02j4h8a39', 'https://ror.org/00v6ky380', 'https://ror.org/01ejq3n45', 'https://ror.org/03w4cze07', 'https://ror.org/013kx3552', 'https://ror.org/01hrdvm22', 'https://ror.org/05b4e2111', 'https://ror.org/037fsex72', 'https://ror.org/02htkw777', 'https://ror.org/003mvv560', 'https://ror.org/022vsje77', 'https://ror.org/04cmvex31', 'https://ror.org/03jbnw407', 'https://ror.org/046sm3c31', 'https://ror.org/01p1gwk51', 'https://ror.org/01aq0vw42', 'https://ror.org/05xgv5563', 'https://ror.org/03aysbj82', 'https://ror.org/02dgz7m30', 'https://ror.org/032vv2x86', 'https://ror.org/05wwhq151', 'https://ror.org/00dkp4z50', 'https://ror.org/042s2nd27', 'https://ror.org/02kj9za59', 'https://ror.org/0015kmr33', 'https://ror.org/007b8s847', 'https://ror.org/01wjwqr61', 'https://ror.org/04af0r679', 'https://ror.org/03b60ep93', 'https://ror.org/04whgpk43', 'https://ror.org/05ns7k858', 'https://ror.org/018hhzz02', 'https://ror.org/00jp3t114', 'https://ror.org/00td6v066', 'https://ror.org/05f0cz467', 'https://ror.org/03x3mrm69', 'https://ror.org/00n36xb70', 'https://ror.org/03k2mbt67', 'https://ror.org/04pexwy87', 'https://ror.org/01kq0a723', 'https://ror.org/04dmckt32', 'https://ror.org/035ysxq78', 'https://ror.org/04c2c2109', 'https://ror.org/01x4jh294']

Proposed solution

Determine if there is an algorithm we can use to select one of the labels as preferred (investigate what ROR themselves do).

Or consider how to implement django-modeltranslation for OrganizationName.value in such a way that tolerates multiple labels for a given language, maybe by creating another OrganizationName object?

Start with src/utils/management/commands/profile_ror_data.py and inspect how ROR records with multiple labels are represented in OrganizationName.value and handled by django-modeltranslation. Research how ROR selects a preferred label, then compare that with the alternative of tolerating multiple labels per language. Done means a documented, agreed approach for selecting or storing translated organization names without losing valid ROR data.