InternLM / InternLM/xtuner

docs or demos to illustrate the key features

Open
#1,061 2 comments 0 reactions 1 assignee Claimed by @pppppM View on GitHub
Dominant language
Python
Stars
5.2k
Forks
448
Avg merge
3d 15h
Merged PRs (30d)
26

Description

Congrats on the release! Some of the features are so cool they feel like black magic.

Would it be possible to explain the key techniques behind those features, or provide a tutorial/demo so users can reproduce the claimed results?

Image

In particularly, I am interested in the claim

> Memory-efficient design: Train 200B MoE models on 64k sequence lengths without sequence parallelism through advanced memory optimization techniques

It sounds very challenging, unless there’s aggressive offloading and recomputation and may suffer from slow iteration speed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.