How to change alignment_mode in a hf pipelines in code?
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
I'm having the same problem described here: https://github.com/microsoft/presidio/issues/1262 - some of the annotations are skipped because of alignment problems between the spaCy pipeline and the hf pipeline wrapper.
I would like to simply substitute one of the components of the spaCy pipeline with a HF model I trained for NER and use it for this task. I'm trying do this using this code:
``` python
import spacy
nlp = spacy.load("en_core_web_sm")
nlp.remove_pipe("ner")
nlp.add_pipe("hf_token_pipe", config={"model": "mikrz/bert-vir_naeus-ner"})
```
The model loads correctly and technically works fine, but some of the tokens are skipped, e.g.:
``` python
text = 'A novel bacteriophage vB_SauS_SA2 (hereafter designated SA2) that infects Staphylococcus aureus was isolated.'
doc = nlp(text)
```
For the code above I'm getting this warning:
``` markdown
spacy_huggingface_pipelines/token_classification.py:129: UserWarning: Skipping annotation, {'entity_group': 'VIR', 'score': 0.67677, 'word': '_ SauS _ SA2', 'start': 24, 'end': 33} is overlapping or can't be aligned for doc 'A novel bacteriophage vB_SauS_SA2 (hereafter designated SA2) that infects Staphylococcus aureus was ...'
warnings.warn(
```
I saw that in https://github.com/microsoft/presidio/issues/1262 someone recommended changing the alignment_mode of the hf pipeline component. How can I do this in code when using the default en_core_web_sm model?
For my use case, I need the spans of the named entities in the form of word numbers, so in the example above it would be:
```
vB_SauS_SA2: (3,3),
Staphylococcus aureus: (11,12)
```
where we get the position the the start and end of the named entitites.
I'd like to change just this part - I would like to avoid training a custom pipeline which, if I understand https://spacy.io/usage/training correctly, seems to be necessary when creating a new spaCy pipeline from the config file. Or did I misunderstand and there is an option to just change parts of the config file? If that's the case, could you instruct me how to create a config file, where to put it and how to use it?
Contributor guide
Assessment
This issue has not been assessed yet.