sillsdev / sillsdev/silnlp

Include testing from the source text(s) where possible.

Open
#375 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39
Forks
7
Avg merge
1d 9h
Merged PRs (30d)
5

Description

Most teams will have a preference for which source text they wish to use for drafting. After we've run alignments, those can be used to inform the team. Some teams will request drafts from more than one source text.
It would be ideal to include a test set in each training run so that we can obtain metrics that more closely reflect the actual performance of the model. These scores will also make for better comparisons between disparate models.

For example, we might train three models, one from a BT, one from a published Bible and one with both (mixed_src). If all three of those contained a test set corpus pair:
corpus_pairs:

  • corpus_books: a small book, or few chapters
    src:
    • src_iso-published_bible
      trg: trg_iso-target_bible
      type: test

It may well be that this is more-or-less already possible and the only change required is to the default config file.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the default config file and the corpus_pairs configuration shown in the issue, then trace how training runs consume entries with type: test. Done means training runs can include a test corpus pair for each requested source text and produce metrics that support comparisons between models.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.