sillsdev / sillsdev/silnlp

Handle "OtherProject" terms specification for biblical terms

Open
#772 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug pipeline 2: extract
Dominant language
Python
Stars
39
Forks
7
Avg merge
1d 9h
Merged PRs (30d)
5

Description

This is related to Issue #661, but separate. The TDKT project refers to a separate list of terms as shown here.

<BiblicalTermsListSetting>OtherProject:TDKT:Hughes(Id)BiblicalTerms.xml</BiblicalTermsListSetting>

This causes the extract_corpora.py script to crash with the following error:

FileNotFoundError: [Errno 2] No such file or directory: 'M:\\MT\\terms\\OtherProject-metadata.txt'

It does produce an extract file, but training fails when using that file, even with key terms training turned off. (Sample ClearML experiment.)

Changing the line in the Settings file to the following (remove "Other") causes the extract script to complete correctly:

<BiblicalTermsListSetting>Project:TDKT:Hughes(Id)BiblicalTerms.xml</BiblicalTermsListSetting>

It does appear that the relevant files are in the project directory, including Hughes(Id)BiblicalTerms.xml, TermRenderings.xml, and ProjectBiblicalTerms.xml. In addition, the following files were created in M:\MT\terms:

  • id-TDKT_2025_07_09-Project-renderings.txt
  • TDKT_2025_07_09-metadata.txt
  • en-TDKT_2025_07_09-glosses.txt
  • TDKT_2025_07_09-vrefs.txt

However, training with this extract file still fails even with terms turned off (see below).

(silnlp) PS C:\Users\betha\silnlp> poetry run python -m silnlp.nmt.experiment Indonesia\Katingan\NLLB.1.3B.id-TDKT.nij-TBNDK --clearml-queue jobs_urgent --save-checkpoints --stats --translate
>>
2025-07-10 10:15:39,943 - silnlp.common.environment - INFO - Using workspace: M:\ as per environment variable SIL_NLP_DATA_PATH.
[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\betha\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
ClearML Task: created new task id=ee38b85e0adf4835be7e09bb0db418b6
ClearML results page: https://app.sil.hosted.allegro.ai/projects/432d9475fffa49d18d85003f956c88a9/experiments/ee38b85e0adf4835be7e09bb0db418b6/output/log
ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
Traceback (most recent call last):
  File "C:\Users\betha\.conda\envs\silnlp\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\betha\.conda\envs\silnlp\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\betha\silnlp\silnlp\nmt\experiment.py", line 215, in <module>
    main()
  File "C:\Users\betha\silnlp\silnlp\nmt\experiment.py", line 194, in main
    exp = SILExperiment(
  File "<string>", line 18, in __init__
  File "C:\Users\betha\silnlp\silnlp\nmt\experiment.py", line 37, in __post_init__
    self.clearml = SILClearML(self.name, self.clearml_queue, commit=self.commit)
  File "<string>", line 10, in __init__
  File "C:\Users\betha\silnlp\silnlp\nmt\clearml_connection.py", line 47, in __post_init__
    self._load_config()
  File "C:\Users\betha\silnlp\silnlp\nmt\clearml_connection.py", line 152, in _load_config
    self.config = create_config(exp_dir, config)
  File "C:\Users\betha\silnlp\silnlp\nmt\config_utils.py", line 21, in create_config
    return HuggingFaceConfig(exp_dir, config)
  File "C:\Users\betha\silnlp\silnlp\nmt\hugging_face_config.py", line 374, in __init__
    super().__init__(exp_dir, config)
  File "C:\Users\betha\silnlp\silnlp\nmt\config.py", line 348, in __init__
    self.corpus_pairs = parse_corpus_pairs(data_config.get("corpus_pairs", []))
  File "C:\Users\betha\silnlp\silnlp\nmt\config.py", line 227, in parse_corpus_pairs
    src_terms_files = get_terms_files(src_files) if is_set(type, DataFileType.TRAIN) else []
  File "C:\Users\betha\silnlp\silnlp\nmt\config.py", line 273, in get_terms_files
    terms_path = get_terms_renderings_path(file.iso, file.project)
  File "C:\Users\betha\silnlp\silnlp\common\corpus.py", line 242, in get_terms_renderings_path
    assert len(matches) == 1
AssertionError

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure in extract_corpora.py with the OtherProject:TDKT setting, then trace how the generated files are handled in silnlp/nmt/config.py and silnlp/common/corpus.py, especially parse_corpus_pairs, get_terms_files, and get_terms_renderings_path. Done means the OtherProject terms setting produces usable files and the reported training configuration no longer crashes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.