explosion / explosion/spaCy

Issues with resuming training in version 3.1.2

Open
#9,192 4 comments 0 reactions 0 assignees View on GitHub
bug training
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

I am trying to continue training an NER model. I have done this in spaCy 3.0.6 in the past numerous times, but after coming back to do it again, I realized that this time it's not working. My current version is 3.1.2.

## My approach to continued training:

To continue training, I basically modified the config file. The 2 components below are changed to the ones following:

### Initial config file setup:

```
[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.tok2vec]
factory = "tok2vec"
```

### Modified config file:

```
[components.ner]
source = "./saved_model/model-last"
#factory = "ner"
#moves = null
#update_with_oracle_cut_size = 100

[components.tok2vec]
source = "./saved_model/model-last"
#factory = "tok2vec"
```

"./saved model" is the address to which the initial training is saved.

I also tried with the following config file setups:

```
[components.ner]
source = "./fake_output_0_backup/model-last"
component = "ner"
#factory = "ner"
#moves = null
#update_with_oracle_cut_size = 100

[components.tok2vec]
source = "./fake_output_0_backup/model-last"
component = "tok2vec"
#factory = "tok2vec"
```

This approach seem to not work anymore as I get the following error:

```
/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/spacy/training/initialize.py:137: UserWarning: [W113] Sourced component 'tok2vec' may not work as expected: source vectors are not identical to current pipeline vectors.
warnings.warn(Warnings.W113.format(name=sourced_component))
/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/spacy/training/initialize.py:137: UserWarning: [W113] Sourced component 'ner' may not work as expected: source vectors are not identical to current pipeline vectors.
warnings.warn(Warnings.W113.format(name=sourced_component))
[2021-09-13 00:01:58,021] [INFO] Finished initializing nlp object
[2021-09-13 00:01:58,021] [INFO] Initialized pipeline components: []
*** Error in `/home/ec2-user/anaconda3/envs/python3/bin/python': corrupted size vs. prev_size: 0x00005617d85ceba0 ***
```

## Question: Has something changed in the recent versions to disable this approach?

I can send the backtrace if it could be helpful, but beware that it's super long and I could not learn the issue from it. :)

I would appreciate any recommendations. I can try in spaCy 3.0.6 if it helps to verify that it's a version issue.

## My Environment

* Operating System: Linux (AWS Sagemaker)
* Python Version Used: 3.6
* spaCy Version Used: 3.1.2

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.