Issues with resuming training in version 3.1.2
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
I am trying to continue training an NER model. I have done this in spaCy 3.0.6 in the past numerous times, but after coming back to do it again, I realized that this time it's not working. My current version is 3.1.2.
## My approach to continued training:
To continue training, I basically modified the config file. The 2 components below are changed to the ones following:
### Initial config file setup:
```
[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100
[components.tok2vec]
factory = "tok2vec"
```
### Modified config file:
```
[components.ner]
source = "./saved_model/model-last"
#factory = "ner"
#moves = null
#update_with_oracle_cut_size = 100
[components.tok2vec]
source = "./saved_model/model-last"
#factory = "tok2vec"
```
"./saved model" is the address to which the initial training is saved.
I also tried with the following config file setups:
```
[components.ner]
source = "./fake_output_0_backup/model-last"
component = "ner"
#factory = "ner"
#moves = null
#update_with_oracle_cut_size = 100
[components.tok2vec]
source = "./fake_output_0_backup/model-last"
component = "tok2vec"
#factory = "tok2vec"
```
This approach seem to not work anymore as I get the following error:
```
/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/spacy/training/initialize.py:137: UserWarning: [W113] Sourced component 'tok2vec' may not work as expected: source vectors are not identical to current pipeline vectors.
warnings.warn(Warnings.W113.format(name=sourced_component))
/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/spacy/training/initialize.py:137: UserWarning: [W113] Sourced component 'ner' may not work as expected: source vectors are not identical to current pipeline vectors.
warnings.warn(Warnings.W113.format(name=sourced_component))
[2021-09-13 00:01:58,021] [INFO] Finished initializing nlp object
[2021-09-13 00:01:58,021] [INFO] Initialized pipeline components: []
*** Error in `/home/ec2-user/anaconda3/envs/python3/bin/python': corrupted size vs. prev_size: 0x00005617d85ceba0 ***
```
## Question: Has something changed in the recent versions to disable this approach?
I can send the backtrace if it could be helpful, but beware that it's super long and I could not learn the issue from it. :)
I would appreciate any recommendations. I can try in spaCy 3.0.6 if it helps to verify that it's a version issue.
## My Environment
* Operating System: Linux (AWS Sagemaker)
* Python Version Used: 3.6
* spaCy Version Used: 3.1.2
Contributor guide
Assessment
This issue has not been assessed yet.