AnswerDotAI / AnswerDotAI/ModernBERT
Missing model_max_length for tokeniser
Open
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
I'm trying to use `Tokenizer` for ModernBERT like this:
```python
model_name = "answerdotai/ModernBERT-large"
tokenizer=AutoTokenizer.from_pretrained(model_name)
print(tokeniser.model_max_length)
# 1000000000000000019884624838656
```
Which gives `model_max_length == int(1e30)`
Shouldn't the `tokenizer` contain a correct max value of 8192?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.