Use a custom exception ValueError subclass for the special tokens warning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
This code here: https://github.com/openai/tiktoken/blob/39f29cecdb6fc38d9a3434e5dd15e4de58cf3c80/tiktoken/core.py#L375-L383
I wanted to do something special on this exception in my own code, so I had to write this:
try:
tokens = encoding.encode(text, **kwargs)
except ValueError as ex:
if 'disallowed special token' in str(ex):
# Do something special
I suggest having a custom exception class for this instead:
class DisallowedSpecialTokenError(ValueError):
pass
Raising that class instead would let people like me catch it explicitly, and since it's a subclass of ValueError it should not break any existing code that currently catches ValueError directly.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked lines in tiktoken/core.py and trace the encode path that raises the warning. Add a named ValueError subclass for disallowed special tokens and verify that callers catching ValueError remain compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100