Clarification on Code Implementation in DeepSeek vs Llama

Dear Authors,

Thank you for providing such excellent work for the community to use!

I have a question regarding an implementation detail. In [Line 338](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Base/blob/ea9b066cee82f82906fdd58898cb3788b1c5d770/modeling_deepseek.py#L338), it appears that the code is adapted from Llama. However, when looking closer, the implementation in DeepSeek seems to differ, particularly from [Line 363](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Base/blob/main/modeling_deepseek.py#L363) to [Line 367](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Base/blob/main/modeling_deepseek.py#L367), compared to Llama’s implementation in [Line 223](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L223).

Could you explain the reasoning behind this difference? Were there specific considerations that led to this change?

I look forward to your response. Thank you again for your great work!

Best regards,