OpenNMT / OpenNMT/CTranslate2

Plans to support "mlp" feature merge embeddings?

Open
#1,112 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
C++
Stars
4.7k
Forks
537
Avg merge
12h 12m
Merged PRs (30d)
4

Description

Tried to convert a transformer model from OpenNMT-py which uses "mlp" for it's embeddings but MLP hasn't been incorporated into conversion, so I assume it's not supported in CTranslate2.

onmt_release_model --model ./saves/model_step_52000.pt --format ctranslate2 --quantization int8 --output ./ct2
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\Scripts\onmt_release_model.exe\__main__.py", line 7, in <module>
  File "C:\Users\admin\AppData\Roaming\Python\Python39\site-packages\onmt\bin\release_model.py", line 30, in main
    converter.convert(opt.output, force=True,
  File "C:\Users\admin\AppData\Roaming\Python\Python39\site-packages\ctranslate2\converters\converter.py", line 89, in convert
    model_spec = self._load()
  File "C:\Users\admin\AppData\Roaming\Python\Python39\site-packages\ctranslate2\converters\opennmt_py.py", line 175, in _load
    return _get_model_spec_seq2seq(
  File "C:\Users\admin\AppData\Roaming\Python\Python39\site-packages\ctranslate2\converters\opennmt_py.py", line 81, in _get_model_spec_seq2seq
    embeddings_merge=_SUPPORTED_FEATURES_MERGE[feat_merge],
KeyError: 'mlp'

Wanted to trial and error mlp against the normal concat to compare and contrast how it may affect results, as they seemed to be promising

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in converters/opennmt_py.py at _SUPPORTED_FEATURES_MERGE and _get_model_spec_seq2seq, where the traceback shows the unsupported "mlp" value is handled. Read how OpenNMT-py represents this embedding merge mode, then verify that conversion succeeds and the resulting model can be used for comparison with concat.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.