Investigate apply_chat_template per message vs. all messages
Open
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
This is an issue to track the investigatory work of #38 .
Before making the repo wide change to use apply_chat_template on all messages vs. per message, we should do some analysis on how frequent the boundary issues are. An example of such analysis would be:
* Take top 25 instruct tokenizers
* Check if there are boundary issues for multiturn (e.g, check `apply_chat_template(messages[1:i]) for i in range(N)`)
* check arbitrary end-strings (ones other than what the instruct checkpoint comes with) assuming users will fine-tune on their new end-string
Contributor guide
Assessment
This issue has not been assessed yet.