Add support for unknown scripts in NLLB
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 3
- Forks
- 2
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 3
Description
The NMT engine does not have good support for scripts that NLLB was not originally trained on. All characters would be added as individual tokens to the vocabulary resulting in poor translation quality. silnlp provides better support for unknown scripts. Here is an example of the silnlp config to enable this support:
tokenizer:
src_vocab_size: 0
trained_tokens: true
trg_vocab_size: 500
update_src: true
update_trg: true
When silnlp is configured in this way, it will train a new subword tokenizer on the target data using the specified trg_vocab_size. The new tokens are added to the NLLB vocabulary. This results in better translation quality for unknown scripts, then the default settings. Similar functionality should be added to the NMT engine. The engine could automatically detect when to enable this feature or it could be enabled using build options.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the NMT engine's tokenizer configuration and the code that builds or updates the NLLB vocabulary. Compare its behavior with the supplied silnlp configuration, especially trained_tokens, update_src, update_trg, and trg_vocab_size. Done means unknown scripts receive subword-tokenizer support, with a documented choice between automatic detection and build-option configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- api, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100