Multitoken sets cannot be matched using quantifier operators
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## Could multitoken sets be matched using quantifier operators?
Good day team,
When matching multiple tokens using a Matcher, it seems advantageous to use quantifier operators, in case the multiple tokens are repeated a variable number of times.
A basic example would be a list of things, where the length of the list can vary. For example "Text with list: red cat, green bat, blue cow.", in this case, multiple tokens are repeated, an adjective, followed by a noun, followed by a punctuation mark, multiple times. Each set of multiple tokens could be matched using `{'POS': 'ADJ'}, {'POS': 'NOUN'}, {'POS': 'PUNCT'}`, but this only matches one single set, and not the full list.
Could these multiple token sets be matched using quantifier operators? Here is a possible way to write a pattern to match the list:
`{'REPEATED_TOKENS': [{'POS': 'ADJ'}, {'POS': 'NOUN'}, {'POS': 'PUNCT'}], 'OP': '{2, 3}'}`
This would match the repeated tokens from 2 to 3 times as given by 'OP'. Here is a complete code example:
```python
import spacy
from spacy.matcher import Matcher
nlp = spacy.load('en_core_web_sm')
doc = nlp("Text with list: red cat, green bat, blue cow.")
matcher = Matcher(nlp.vocab)
matcher.add('rule1', [[{'REPEATED_TOKENS': [{'POS': 'ADJ'}, {'POS': 'NOUN'}, {'POS': 'PUNCT'}], 'OP': '{2, 3}'}]])
coincidences = matcher(doc)
```
This would match the full list "red cat, green bat, blue cow." in the `coincidences`.
Could this be possible in the future? I am aware of the workaround to achieve this using extensions, but ideally this could be done using native syntax to create patterns to match.
Thank you very much for your attention 🙏
## Your Environment
* Operating System: macOS-26.6.2-arm64-arm-64bit
* Python Version Used: 3.12.12
* spaCy Version Used: 3.8.16
* Environment Information:
Contributor guide
Research direction
Start with the spaCy Matcher API and the pattern syntax shown in the issue, then compare it with the mentioned extensions workaround. Determine how repeated multitoken sets should be represented and matched, including the `{2, 3}` quantifier case; done means the proposed native syntax works for the complete example list.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100