Clarification Needed on Utillization of Tokenization in the Fine-Tuning Module || InternLM-XComposer2d5

Hello Fellow Developers,

I am working on implementing the evaluation code in the current fine-tuning module and noticed something regarding the tokenizer.

While the tokenizer is passed into the make_supervised_data_module function, it doesn't seem to be utilized in the DataCollatorForSupervisedDataset.

**Since DataCollatorForSupervisedDataset serves as the custom data collator, if the tokenizer isn’t used there, what is being employed for tokenization? This brings up the concern of whether the fine-tuning script is functioning as intended.**

Could you please clarify this?

**> Also, when are you planning to release the evaluation code?**

Thanks in Advance.