[TRACKER] Customer support related PR tracker for Intel devices

This issue acted as a PR tracker to Intel customer support related PRs. The purpose is to get understanding of what each PR does and how important are they compared to other customer support related PRs. This also help us to aware of merged PRs and PRs progress.

Under review

• sequence parallel for uneven heads: https://github.com/microsoft/DeepSpeed/pull/6392 (Open)
• Enabled configurable auto Tensor Parallelism (TP) for the inference of diverse models: https://github.com/microsoft/DeepSpeed/pull/6553 (Open)

Already merged

o MoE

• support bf16_optimizer moe expert parallel training and moe EP grad_scale/grad_norm fix: https://github.com/microsoft/DeepSpeed/pull/5259
• Fix a convergence issues in TP topology caused by incorrect grad_norm: https://github.com/microsoft/DeepSpeed/pull/5411
• add moe topk(k>2) gate support: https://github.com/microsoft/DeepSpeed/pull/5881
• reduce cpu host overhead when using moe: https://github.com/microsoft/DeepSpeed/pull/5578

o Ulysess

• fix sequence parallel(Ulysses) grad scale for zero0: https://github.com/microsoft/DeepSpeed/pull/5555
• sequence parallel with communication overlap: https://github.com/microsoft/DeepSpeed/pull/5691

o AutoTP

• autoTP for fused qkv weight: https://github.com/microsoft/DeepSpeed/pull/3844
• autoTP for Qwen: https://github.com/microsoft/DeepSpeed/pull/4902
• Enabled Qwen2-MoE Tensor Parallelism (TP) inference: https://github.com/microsoft/DeepSpeed/pull/6551 (Open)

o Accelerator Graph

• Capture short kernel sequences to graph: https://github.com/microsoft/DeepSpeed/pull/4318

o ZeRO

• params partition for skip_init: https://github.com/microsoft/DeepSpeed/pull/4722

o Others

• skip bcast when enable pp but pp_group_size=1: https://github.com/microsoft/DeepSpeed/pull/3915
• remove duplicate check for pp and zero stage: https://github.com/microsoft/DeepSpeed/pull/4033
• update ut/doc for glm/codegen: https://github.com/microsoft/DeepSpeed/pull/4057
• do allgather only in shared optimizer states groups: https://github.com/microsoft/DeepSpeed/pull/4167
• use non_reentrant_checkpoint fix requires_grad of input must be true for activation checkpoint layer in pipeline train.: https://github.com/microsoft/DeepSpeed/pull/4224
• clear redundant parameters in zero3 bwd hook: https://github.com/microsoft/DeepSpeed/pull/4520
• set the default to use set_to_none for clearing gradients in BF16 optimizer.: https://github.com/microsoft/DeepSpeed/pull/5434
• Use deepspeed.comm instead of torch.distributed: https://github.com/microsoft/DeepSpeed/pull/5225
• Use torch.nan_to_num replace numpy wrapper one: https://github.com/microsoft/DeepSpeed/pull/5877
• [bugfix] promote state in bf16_optimizer: https://github.com/microsoft/DeepSpeed/pull/5767

This is a PR tracker and names no source files, tests, or implementation entry points. Start by reviewing the linked PRs 6392 and 6553 and the older entries; the tracker is done when each listed PR has an accurate status and the support-related work is clearly summarized.