Less is more for alignment (LIMA) - adding special EOT token

Hi,

any help or guidance on how to add a special EOT token, as described in the LIMA paper by Meta?

More specifically, in section 3, Training LIMA, they describe the following:

To differentiate between each speaker (user and assistant), we introduce a special end-of-turn token (EOT) at the end
of each utterance; this token plays the same role as EOS of halting generation, but avoids conflation
with any other meaning that the pretrained model may have imbued into the preexisting EOS token

Thanks!

The issue does not name files, tests, or entry points. Start by locating the tokenization and training entry points in this Python LLaMA implementation, then determine how the LIMA EOT token should be represented and used between utterances. Done would require documented behavior and verification, but the issue does not define a concrete acceptance test.