Extremely Long Text results in PanicException, which is hard to catch in python code
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
For some extremely long sequences, the tokenizer can result in a PanicException. Example
import tiktoken
tokenizer = tiktoken.get_encoding("cl100k_base")
text = "^" * 1000000
tokenizer.encode(text) # this throws a PanicException
The issue is that PanicException is not caught even while catching Exception, and can only be caught by catching a BaseException, which is too broad.
Would it be possible to raise a better exception for such a scenario (maybe something similar to what was done here ?)
The workaround that I currently have is catching the BaseException, and checking for "PanicException" in the exception message. Not sure if it is the best way to do this. Would be grateful for any guidance :)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the tokenizer.encode call in the provided Python reproduction and test the million-character input that triggers PanicException. Trace how that failure is exposed to Python, then verify that the resulting exception can be caught with Exception rather than requiring BaseException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100