huggingface / huggingface/alignment-handbook
Tokenizer model_max_length
Open
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 490
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
Hello,
I was seeing warning during finetuning Mistral and tracked this line here
https://github.com/huggingface/alignment-handbook/blob/main/src/alignment/model_utils.py#L71
Because Mistral's tokenizer model max length has a large number so the model_max_length set as 2048. However my training data consists sequence length longer than that, e.g. 4000 characters. Would this be a problem?
Thank you!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.