Handle "OtherProject" terms specification for biblical terms
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39
- Forks
- 7
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 5
Description
This is related to Issue #661, but separate. The TDKT project refers to a separate list of terms as shown here.
<BiblicalTermsListSetting>OtherProject:TDKT:Hughes(Id)BiblicalTerms.xml</BiblicalTermsListSetting>
This causes the extract_corpora.py script to crash with the following error:
FileNotFoundError: [Errno 2] No such file or directory: 'M:\\MT\\terms\\OtherProject-metadata.txt'
It does produce an extract file, but training fails when using that file, even with key terms training turned off. (Sample ClearML experiment.)
Changing the line in the Settings file to the following (remove "Other") causes the extract script to complete correctly:
<BiblicalTermsListSetting>Project:TDKT:Hughes(Id)BiblicalTerms.xml</BiblicalTermsListSetting>
It does appear that the relevant files are in the project directory, including Hughes(Id)BiblicalTerms.xml, TermRenderings.xml, and ProjectBiblicalTerms.xml. In addition, the following files were created in M:\MT\terms:
- id-TDKT_2025_07_09-Project-renderings.txt
- TDKT_2025_07_09-metadata.txt
- en-TDKT_2025_07_09-glosses.txt
- TDKT_2025_07_09-vrefs.txt
However, training with this extract file still fails even with terms turned off (see below).
(silnlp) PS C:\Users\betha\silnlp> poetry run python -m silnlp.nmt.experiment Indonesia\Katingan\NLLB.1.3B.id-TDKT.nij-TBNDK --clearml-queue jobs_urgent --save-checkpoints --stats --translate
>>
2025-07-10 10:15:39,943 - silnlp.common.environment - INFO - Using workspace: M:\ as per environment variable SIL_NLP_DATA_PATH.
[nltk_data] Downloading package punkt to
[nltk_data] C:\Users\betha\AppData\Roaming\nltk_data...
[nltk_data] Package punkt is already up-to-date!
ClearML Task: created new task id=ee38b85e0adf4835be7e09bb0db418b6
ClearML results page: https://app.sil.hosted.allegro.ai/projects/432d9475fffa49d18d85003f956c88a9/experiments/ee38b85e0adf4835be7e09bb0db418b6/output/log
ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
Traceback (most recent call last):
File "C:\Users\betha\.conda\envs\silnlp\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\betha\.conda\envs\silnlp\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\betha\silnlp\silnlp\nmt\experiment.py", line 215, in <module>
main()
File "C:\Users\betha\silnlp\silnlp\nmt\experiment.py", line 194, in main
exp = SILExperiment(
File "<string>", line 18, in __init__
File "C:\Users\betha\silnlp\silnlp\nmt\experiment.py", line 37, in __post_init__
self.clearml = SILClearML(self.name, self.clearml_queue, commit=self.commit)
File "<string>", line 10, in __init__
File "C:\Users\betha\silnlp\silnlp\nmt\clearml_connection.py", line 47, in __post_init__
self._load_config()
File "C:\Users\betha\silnlp\silnlp\nmt\clearml_connection.py", line 152, in _load_config
self.config = create_config(exp_dir, config)
File "C:\Users\betha\silnlp\silnlp\nmt\config_utils.py", line 21, in create_config
return HuggingFaceConfig(exp_dir, config)
File "C:\Users\betha\silnlp\silnlp\nmt\hugging_face_config.py", line 374, in __init__
super().__init__(exp_dir, config)
File "C:\Users\betha\silnlp\silnlp\nmt\config.py", line 348, in __init__
self.corpus_pairs = parse_corpus_pairs(data_config.get("corpus_pairs", []))
File "C:\Users\betha\silnlp\silnlp\nmt\config.py", line 227, in parse_corpus_pairs
src_terms_files = get_terms_files(src_files) if is_set(type, DataFileType.TRAIN) else []
File "C:\Users\betha\silnlp\silnlp\nmt\config.py", line 273, in get_terms_files
terms_path = get_terms_renderings_path(file.iso, file.project)
File "C:\Users\betha\silnlp\silnlp\common\corpus.py", line 242, in get_terms_renderings_path
assert len(matches) == 1
AssertionError
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure in extract_corpora.py with the OtherProject:TDKT setting, then trace how the generated files are handled in silnlp/nmt/config.py and silnlp/common/corpus.py, especially parse_corpus_pairs, get_terms_files, and get_terms_renderings_path. Done means the OtherProject terms setting produces usable files and the reported training configuration no longer crashes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100