deepspeedai / deepspeedai/DeepSpeed

[BUG]DeepCompile with zero1 has LoweringException

Open
#7,522 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Describe the bug
Using DeepCompile with zero1_compile yield error like:

Launching compile passes: global_steps=0 passes=[<function add_z1_reduce at 0x7fca143c0310>]
[rank0]:   File "/home/miniconda3/envs/dl/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1268, in call_function
[rank0]:     raise LoweringException(e, target, args, kwargs).with_traceback(
[rank0]:   File "/home/miniconda3/envs/dl/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1258, in call_function
[rank0]:     out = lowerings[target](*args, **kwargs)  # type: ignore[index]
[rank0]:   File "/home/miniconda3/envs/dl/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 446, in wrapped
[rank0]:     out = decomp_fn(*args, **kwargs)
[rank0]:   File "/home/miniconda3/envs/dl/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 2156, in randn
[rank0]:     raise AssertionError("should have been handled in replace_random.py")
[rank0]: torch._inductor.exc.InductorError: LoweringException: AssertionError: should have been handled in replace_random.py
[rank0]:   target: aten.randn.default
[rank0]:   args[0]: [48, 722]
[rank0]:   kwargs: {'device': device(type='cuda', index=0), 'pin_memory': False}

To Reproduce

def make_schedule(passes, warmup):
    from deepspeed.compile.passes import zero1_compile, prefetch, selective_gather

    schedule = []

    schedule.append((0, [zero1_compile.add_z1_reduce]))
    second_opt = [zero1_compile.add_z1_reduce]
    if "prefetch" in passes:
        second_opt.append(prefetch.schedule_prefetch)
    if "selective_gather" in passes:
        second_opt.append(selective_gather.selective_gather)
    schedule.append((warmup, second_opt))
    return schedule

schedule = make_schedule(["prefetch", "selective_gather"], warmup=5)
model.compile(schedule=schedule)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the schedule using deepspeed.compile.passes.zero1_compile.add_z1_reduce with the prefetch and selective_gather passes, then inspect how the pass interacts with the reported aten.randn.default lowering failure in PyTorch Inductor. Done means DeepCompile with zero1_compile no longer raises the LoweringException during compilation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.