huggingface / huggingface/transformers

Key Error: 'pre-processing' during conversion from tatoeba to Marian model

Open
#11,647 3 comments 1 reaction 1 assignee Claimed by @patil-suraj View on GitHub
WIP
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 8h
Merged PRs (30d)
276

Description

## Environment info

- `transformers` version: `4.6.0.dev0`
- Platform: `CentOS Linux release 7.7.1908 (Core)`
- Python version: `3.8.5`
- PyTorch version: `1.8.1 + cuda 10.2`
- Tensorflow version: N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Marian: @patrickvonplaten , @patil-suraj

## Information

Model I am using (Bert, XLNet ...): Marian

The problem arises when using:
* [x] the official example scripts: tatoeba to marian model script
* [ ] my own modified scripts

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: machine translation
* [ ] my own task or dataset

## To reproduce

Following the script from [scripts/tatoeba/README.md ](https://github.com/huggingface/transformers/tree/master/scripts/tatoeba)
1.
```git clone git@github.com:huggingface/transformers.git
cd transformers
pip install -e .
pip install pandas GitPython wget
```
2.
```
curl https://cdn-datasets.huggingface.co/language_codes/language-codes-3b2.csv > language-codes-3b2.csv
curl https://cdn-datasets.huggingface.co/language_codes/iso-639-3.csv > iso-639-3.csv
```
3. `git clone git@github.com:Helsinki-NLP/Tatoeba-Challenge.git`
4. `python src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py --models kor-eng eng-kor --save_dir converted/`

Error message:
```
Traceback (most recent call last):
File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 1267, in
resolver = TatoebaConverter(save_dir=args.save_dir)
File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 58, in __init__
reg = self.make_tatoeba_registry()
File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 258, in make_tatoeba_registry
return [(k, v["pre-processing"], v["download"], v["download"][:-4] + ".test.txt") for k, v in results.items()]
File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 258, in
return [(k, v["pre-processing"], v["download"], v["download"][:-4] + ".test.txt") for k, v in results.items()]
KeyError: 'pre-processing'
```

## Expected behavior

Conversion of the model from Tatoeba to Marian for the chosen language pair with no errors.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.