[QUESTION] Why Change Special Tokens ID After Training DeepSpeedChat

Hello again, I wish you good work.

My question is, this is the special tokens id for load_hf_tokenizer, which is the first version of our model (OPT-1.3b etc). As follows :

{'bos_token': '</s>',
  'eos_token': '</s>',
  'unk_token': '</s>',
  'pad_token': '</s>'}


But when we call tokenizer with load_hf_tokenizer after training, the special tokens id changes. As follows :

{'bos_token': '<|endoftext|>',
  'eos_token': '<|endoftext|>',
  'unk_token': '<|endoftext|>',
  'pad_token': '<|endoftext|>'}


What is the reason for this change? I would be glad if you could help me, I wish you good work.

@HeyangQin @awan-10 @yaozhewei

Start by tracing how load_hf_tokenizer is configured before and after DeepSpeedChat training, using the OPT-1.3b tokenizer values shown in the issue. Compare the saved tokenizer configuration and explain why the special tokens change from to <|endoftext|>.