deepspeedai / deepspeedai/DeepSpeed

How to support topk>2 (topk=6 is needed in our experiment) in MoE model?

Open
#5,155 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

we find that

@staticmethod
def supports_config(config: DSMoEConfig) -> bool:
if config.input_dtype != config.output_dtype:
return False

    if config.input_dtype != torch.float16 and config.input_dtype != torch.bfloat16:
        return False

    if config.top_k != 1 and config.top_k != 2:
        return False

    return True

in "deepspeed.inference.v2.modules.implementations.moe.cutlass_multi_gemm.DSMultiGemmMoE"

It means DSMoE only supoort topk=1 or topk =2.

But our experiment need support for topk=6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at deepspeed.inference.v2.modules.implementations.moe.cutlass_multi_gemm.DSMultiGemmMoE.supports_config and trace how top_k is handled by the MoE implementation. Determine the existing validation and execution constraints for values above 2; done means top_k=6 is supported without breaking the current top_k=1 and top_k=2 cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.