InternLM / InternLM/lmdeploy

[Feature] change InternLM2 modeling to unified type

Open
#1,224 1 comment 0 reactions 1 assignee Claimed by @HIT-cwh View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Motivation

when do the w8a8 quantization in pytorch engine, I found that InternLM2 modeling like. It use self.attention, self.feed_forward...
```python

class InternLM2DecoderLayer(nn.Module):
  def __init__(self, config: InternLM2Config):
  super().__init__()
  self.hidden_size = config.hidden_size
   
  self.attention = INTERNLM2_ATTENTION_CLASSES[config.attn_implementation](config=config)
   
  self.feed_forward = InternLM2MLP(config)
  self.attention_norm = InternLM2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
  self.ffn_norm = InternLM2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
```
but others model modeling like, and all use self.self_attn, self.mlp, they have the diff name.
```python
class LlamaDecoderLayer(nn.Module):
"""Decoder layer for Llama Model."""

def __init__(self, config: LlamaConfig):
super().__init__()
self.hidden_size = config.hidden_size
self.self_attn = LlamaAttention(config=config)
self.mlp = LlamaMLP(config)
self.input_layernorm = LlamaRMSNorm(config.hidden_size,
eps=config.rms_norm_eps)
self.post_attention_layernorm = LlamaRMSNorm(config.hidden_size,
eps=config.rms_norm_eps)
```
using unifiy name is important for quantization, and for other feature like Medusa, or Sparse Gemm(they need to change self.mlp layer).
**If we do not unify the names, we would need to continuously iterate through class names to find the layers that need to be quantized or replaced. This makes the code difficult to understand and also results in additional time consumption.**
> and if necessary, I will fix it later...

### Related resources

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.