explosion / explosion/spaCy

How to change alignment_mode in a hf pipelines in code?

Open
#13,548 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

I'm having the same problem described here: https://github.com/microsoft/presidio/issues/1262 - some of the annotations are skipped because of alignment problems between the spaCy pipeline and the hf pipeline wrapper.

I would like to simply substitute one of the components of the spaCy pipeline with a HF model I trained for NER and use it for this task. I'm trying do this using this code:
``` python
import spacy
nlp = spacy.load("en_core_web_sm")
nlp.remove_pipe("ner")

nlp.add_pipe("hf_token_pipe", config={"model": "mikrz/bert-vir_naeus-ner"})
```

The model loads correctly and technically works fine, but some of the tokens are skipped, e.g.:
``` python
text = 'A novel bacteriophage vB_SauS_SA2 (hereafter designated SA2) that infects Staphylococcus aureus was isolated.'
doc = nlp(text)
```
For the code above I'm getting this warning:
``` markdown
spacy_huggingface_pipelines/token_classification.py:129: UserWarning: Skipping annotation, {'entity_group': 'VIR', 'score': 0.67677, 'word': '_ SauS _ SA2', 'start': 24, 'end': 33} is overlapping or can't be aligned for doc 'A novel bacteriophage vB_SauS_SA2 (hereafter designated SA2) that infects Staphylococcus aureus was ...'
warnings.warn(
```

I saw that in https://github.com/microsoft/presidio/issues/1262 someone recommended changing the alignment_mode of the hf pipeline component. How can I do this in code when using the default en_core_web_sm model?
For my use case, I need the spans of the named entities in the form of word numbers, so in the example above it would be:
```
vB_SauS_SA2: (3,3),
Staphylococcus aureus: (11,12)
```
where we get the position the the start and end of the named entitites.

I'd like to change just this part - I would like to avoid training a custom pipeline which, if I understand https://spacy.io/usage/training correctly, seems to be necessary when creating a new spaCy pipeline from the config file. Or did I misunderstand and there is an option to just change parts of the config file? If that's the case, could you instruct me how to create a config file, where to put it and how to use it?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.