huggingface / huggingface/transformers
Key Error: 'pre-processing' during conversion from tatoeba to Marian model
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 276
Description
## Environment info
- `transformers` version: `4.6.0.dev0`
- Platform: `CentOS Linux release 7.7.1908 (Core)`
- Python version: `3.8.5`
- PyTorch version: `1.8.1 + cuda 10.2`
- Tensorflow version: N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
### Who can help
Marian: @patrickvonplaten , @patil-suraj
## Information
Model I am using (Bert, XLNet ...): Marian
The problem arises when using:
* [x] the official example scripts: tatoeba to marian model script
* [ ] my own modified scripts
The tasks I am working on is:
* [x] an official GLUE/SQUaD task: machine translation
* [ ] my own task or dataset
## To reproduce
Following the script from [scripts/tatoeba/README.md ](https://github.com/huggingface/transformers/tree/master/scripts/tatoeba)
1.
```git clone git@github.com:huggingface/transformers.git
cd transformers
pip install -e .
pip install pandas GitPython wget
```
2.
```
curl https://cdn-datasets.huggingface.co/language_codes/language-codes-3b2.csv > language-codes-3b2.csv
curl https://cdn-datasets.huggingface.co/language_codes/iso-639-3.csv > iso-639-3.csv
```
3. `git clone git@github.com:Helsinki-NLP/Tatoeba-Challenge.git`
4. `python src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py --models kor-eng eng-kor --save_dir converted/`
Error message:
```
Traceback (most recent call last):
File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 1267, in
resolver = TatoebaConverter(save_dir=args.save_dir)
File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 58, in __init__
reg = self.make_tatoeba_registry()
File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 258, in make_tatoeba_registry
return [(k, v["pre-processing"], v["download"], v["download"][:-4] + ".test.txt") for k, v in results.items()]
File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 258, in
return [(k, v["pre-processing"], v["download"], v["download"][:-4] + ".test.txt") for k, v in results.items()]
KeyError: 'pre-processing'
```
## Expected behavior
Conversion of the model from Tatoeba to Marian for the chosen language pair with no errors.
Contributor guide
Assessment
This issue has not been assessed yet.