Exception when parsing ' ' with tokenize=False
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.9k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
This throws an exception with pattern 2.6:
from pattern.en import parse
result = parse(' ', tokenize=False)
The last frame of the exception:
/pattern/text/__init__.pyc in _v(self, token, previous, next)
310 v = {}
311 f(v, "b", "b") # Bias.
--> 312 f(v, "h", token[0]) # Capitalization.
313 f(v, "w", token[-6:] if token not in self.known or token in self.unknown else "")
314 f(v, "x", token[-3:]) # Word suffix.
IndexError: string index out of range
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the exception with pattern.en.parse(' ', tokenize=False), then inspect pattern/text/init.py around the _v method and the call path from parse. Determine the expected handling for a space-only input and add a regression test; done means the example no longer raises IndexError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100