sillsdev / sillsdev/serval

Add support for unknown scripts in NLLB

Open
#807 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
3
Forks
2
Avg merge
3d 1h
Merged PRs (30d)
3

Description

The NMT engine does not have good support for scripts that NLLB was not originally trained on. All characters would be added as individual tokens to the vocabulary resulting in poor translation quality. silnlp provides better support for unknown scripts. Here is an example of the silnlp config to enable this support:

  tokenizer:
    src_vocab_size: 0
    trained_tokens: true
    trg_vocab_size: 500
    update_src: true
    update_trg: true

When silnlp is configured in this way, it will train a new subword tokenizer on the target data using the specified trg_vocab_size. The new tokens are added to the NLLB vocabulary. This results in better translation quality for unknown scripts, then the default settings. Similar functionality should be added to the NMT engine. The engine could automatically detect when to enable this feature or it could be enabled using build options.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the NMT engine's tokenizer configuration and the code that builds or updates the NLLB vocabulary. Compare its behavior with the supplied silnlp configuration, especially trained_tokens, update_src, update_trg, and trg_vocab_size. Done means unknown scripts receive subword-tokenizer support, with a documented choice between automatic detection and build-option configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
api, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.