deepspeedai / deepspeedai/DeepSpeed
[BUG]DeepCompile with zero1 has LoweringException
Open
Nobody has claimed this yet.
bug
training
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
Using DeepCompile with zero1_compile yield error like:
Launching compile passes: global_steps=0 passes=[<function add_z1_reduce at 0x7fca143c0310>]
[rank0]: File "/home/miniconda3/envs/dl/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1268, in call_function
[rank0]: raise LoweringException(e, target, args, kwargs).with_traceback(
[rank0]: File "/home/miniconda3/envs/dl/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1258, in call_function
[rank0]: out = lowerings[target](*args, **kwargs) # type: ignore[index]
[rank0]: File "/home/miniconda3/envs/dl/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 446, in wrapped
[rank0]: out = decomp_fn(*args, **kwargs)
[rank0]: File "/home/miniconda3/envs/dl/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 2156, in randn
[rank0]: raise AssertionError("should have been handled in replace_random.py")
[rank0]: torch._inductor.exc.InductorError: LoweringException: AssertionError: should have been handled in replace_random.py
[rank0]: target: aten.randn.default
[rank0]: args[0]: [48, 722]
[rank0]: kwargs: {'device': device(type='cuda', index=0), 'pin_memory': False}
To Reproduce
def make_schedule(passes, warmup):
from deepspeed.compile.passes import zero1_compile, prefetch, selective_gather
schedule = []
schedule.append((0, [zero1_compile.add_z1_reduce]))
second_opt = [zero1_compile.add_z1_reduce]
if "prefetch" in passes:
second_opt.append(prefetch.schedule_prefetch)
if "selective_gather" in passes:
second_opt.append(selective_gather.selective_gather)
schedule.append((warmup, second_opt))
return schedule
schedule = make_schedule(["prefetch", "selective_gather"], warmup=5)
model.compile(schedule=schedule)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the schedule using deepspeed.compile.passes.zero1_compile.add_z1_reduce with the prefetch and selective_gather passes, then inspect how the pass interacts with the reported aten.randn.default lowering failure in PyTorch Inductor. Done means DeepCompile with zero1_compile no longer raises the LoweringException during compilation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100