InternLM / InternLM/lmdeploy

[Feature] Load dataset from ModelScope during quantization

Open
#1,144 2 comments 0 reactions 1 assignee Claimed by @pppppM View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Motivation

由于量化时需要 load calib 数据集,而目前默认是从 hf 进行 load,这可能会对无法连接到 hf 的用户造成一定的困扰。

因此,能否加入从 ModelScope load 相关数据集的逻辑?

### Related resources

_No response_

### Additional context

以下是个人一些不成熟的 idea:

对于 https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/lite/utils/calib_dataloader.py 文件,以 c4 数据集为例,#L93-#L105 段为从 hf load 逻辑。修改后大致如下:

```python
try:
# load dataset from hf
is_loaded = True
catch Exception:
is_loaded = False
# logger.warning: Cannot load xxx from hf

if not is_loaded:
try:
# load from ms
is_loaded = True
catch Exception:
is_loaded = False
# logger.warning: Cannot load xxx from ms

if not is_loaded:
raise Exception

...
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.