huggingface / huggingface/optimum-intel

OVModelForSeq2SeqLM with Helsinki-NLP/opus-mt-es-en has slow inference times when exported to OpenVino

Open
#339 2 comments 0 reactions 1 assignee Claimed by @echarlaix View on GitHub
Dominant language
Jupyter Notebook
Stars
620
Forks
270
Avg merge
3d 10h
Merged PRs (30d)
22

Description

I'm having trouble exporting the `Helsinki-NLP/opus-mt-es-en` model for language translation into the optimised OpenVino IR format. Reading through the other issues within this repository highlighted this issue https://github.com/huggingface/optimum-intel/issues/188, which seems to suffer from similar effects.

In that case, it seemed to be an issue with the BigBird architecture and its lack of support by HuggingFace Optimum. However, the `Helsinki-NLP/opus-mt-es-en` model is of the [`MarianMT` class](https://huggingface.co/Helsinki-NLP/opus-mt-es-en/blob/main/config.json#L8), which is [documented](https://huggingface.co/docs/optimum/bettertransformer/overview#supported-models) as being supported.

Am I missing something here fundamental? Is the conversion of the `MarianMT` model into OpenVino IR format currently unsupported by this library in a similar way to the BigBird models as in the above issue? Or are there aspects of the conversion that I am not specifying correctly such that the export is sub-optimal? It would seem that this should be possible given the [documentation](https://huggingface.co/docs/optimum/intel/inference#export-and-inference-of-sequencetosequence-models).

I see the following during the build logs if it helps at all: `Asked a sequence length of 16, but a sequence length of 1 will be used with use_past == True for 'decoder_input_ids'.`

An MRE looks like:
```python
import os
from optimum.intel.openvino import OVModelForSeq2SeqLM
from transformers import AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-es-en")
ov_model = OVModelForSeq2SeqLM.from_pretrained(
"Helsinki-NLP/opus-mt-es-en",
export=True,
use_cache=True,
)

def run(text: str):
pipe = pipeline("translation_es_to_en", model=ov_model, tokenizer=tokenizer)
return pipe(text)

def export_to_ov(save_dir: str):
ov_model.save_pretrained()

if __name__ == "__main__":
export_to_ov("./")
```

Operating `run("Hola, como estas?")` yields an inference time of `0.6323761940002441s` while using the exported OVIR binaries in an OVMS model pipeline yields an inference time of `45s`.

Any help on this one would be greatly appreciated, cheers!

P.S. I can post the `config.json` file being passed to the OVMS instance, but it's very long so I'll leave it until it's required!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.