deepspeedai / deepspeedai/DeepSpeed
[TASK] Seperate AutoTP workflow
Open
@mrwyattii is already working on this.
Since Jan 9, 2024.
enhancement
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
As discussed in this PR (https://github.com/microsoft/DeepSpeed/pull/4721), we need to increase test coverage for AutoTP to cover more models. Such workflow can help avoid regressions such as https://github.com/microsoft/DeepSpeed/issues/4774
This is a challenge in current UT scope because of the following points:
- Popular models has very large model checkpoints (~6B to ~180B), we need an instance large enough to be able to download and run these large models.
- To test effectiveness of AutoTP, certain metric i.e. accuracy or perplexity will be needed to verify the effectiveness of AutoTP
- The workflow needs to be expandable to new model supported by DeepSpeed.
The workflow may also run the following variants:
- load checkpoint with from_config as memory efficient form.
- Quantization form of the model.
- 3 devices to test uneven sharding.
The expected result of this task is:
- A workflow that can regularly test the AutoTP status of each model and post test result. (pass/fail, accuracy, etc.) This can complement the manually maintained list (https://github.com/microsoft/DeepSpeed/blob/master/docs/_tutorials/automatic-tensor-parallelism.md) which is often out of sync.
- A test script that people can use to reproduce and report AutoTP related issues.
- Better integration process of new model AutoTP support (what was broken, which PR fixed it, etc.)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.