facebookresearch / facebookresearch/CodeGen

Training MLM with reload model as TransCoder_model_1 on csharp monolingual data generated from pre_processing

Open
#90 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
777
Forks
144
PR merge metrics
No merged PRs in 30d

Description

We have pre-processed our data in csharp and ruby, and obtained their monolingual data and after that we are trying to run the MLM step by reloading TransCoder_model_1, but in the logs we see these warnings and also the accuracy for the run is pretty low and the training stops after 10 epochs as there is no improvement. Then, we've tried givign only csharp as the language and it's the same. Is the below behaviour expected while training?

```
INFO - 04/05/23 15:15:35 - 0:00:05 - ============ Model Reloading
INFO - 04/05/23 15:15:35 - 0:00:05 - Reloading encoder from TransCoder_model_1.pth ...
WARNING - 04/05/23 15:15:43 - 0:00:13 - When reloading word embeddings, could not find embeddings for 29436 words: ['CLASS_0', 'CLASS_1', 'CLASS_2', 'CLASS_3', 'CLASS_4', '...', '経', '者の@@', '要素の@@', '見た@@', '鉛@@']... Initializing them to < unk >.
WARNING - 04/05/23 15:15:44 - 0:00:14 - No match found for lang csharp_monolingual csharp_monolingual in dict_keys(['cpp_sa', 'java_sa', 'python_sa']). Initializing randomly.
WARNING - 04/05/23 15:15:44 - 0:00:14 - The size of position embeddings in current model is 2048, the size of reloaded is 1024. need to repeat last positions 1024 times.
INFO - 04/05/23 15:15:44 - 0:00:14 - Reloading decoders from TransCoder_model_1.pth ...
WARNING - 04/05/23 15:15:44 - 0:00:15 - When reloading word embeddings, could not find embeddings for 29436 words: ['CLASS_0', 'CLASS_1', 'CLASS_2', 'CLASS_3', 'CLASS_4', '...', '経', '者の@@', '要素の@@', '見た@@', '鉛@@']... Initializing them to < unk >.
WARNING - 04/05/23 15:15:45 - 0:00:15 - No match found for lang csharp_monolingual csharp_monolingual in dict_keys(['cpp_sa', 'java_sa', 'python_sa']). Initializing randomly.
WARNING - 04/05/23 15:15:45 - 0:00:15 - The size of position embeddings in current model is 2048, the size of reloaded is 1024. need to repeat last positions 1024 times.
WARNING - 04/05/23 15:15:45 - 0:00:15 - Parameter layer_norm15.0.weight not found.
WARNING - 04/05/23 15:15:45 - 0:00:15 - Parameter layer_norm15.0.bias not found.
WARNING - 04/05/23 15:15:45 - 0:00:15 - Parameter encoder_attn.0.q_lin.weight not found.
WARNING - 04/05/23 15:15:45 - 0:00:15 - Parameter encoder_attn.0.q_lin.bias not found.
WARNING - 04/05/23 15:15:45 - 0:00:15 - Parameter encoder_attn.0.k_lin.weight not found.
WARNING - 04/05/23 15:15:45 - 0:00:15 - Parameter encoder_attn.0.k_lin.bias not found.
WARNING - 04/05/23 15:15:45 - 0:00:15 - Parameter encoder_attn.0.v_lin.weight not found.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.