mlcommons / mlcommons/chakra

chakra_converter PyTorch - conversion failed to due to Cyclic dependecny in DAG

Open
#189 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
196
Forks
84
PR merge metrics
No merged PRs in 30d

Description

Describe the Bug

While converting the ET+ pytroch trace to Protobuf format facing conversion error due Cyclic dependency detected.

Steps to Reproduce

Run pytorch resnet18 model with execution trace enable to get both host and device traces.
Use chakra trace link to merge the traces.
use chakra converter to convert to PB.

chakra_converter PyTorch --input chakra_host_device_trace_0.json --output chakra_
host_device_trace_0.pb

chakra repo
commit bf222b23caf7b6972da17ab7b419d02083c2d2e0 (HEAD -> main, origin/main, origin/HEAD)

Expected Behavior

file in Protobuf formation.

ERROR [03/23/2025 09:28:44 AM] Cyclic dependency detected: aten::lift_fresh -> aten::result_type -> aten::randint -> aten::sub_ -> aten::sub_ -> aten::view -> aten::view -> aten::item -> aten::is_nonzero -> aten::as_strided -> aten::any -> aten::to -> aten::eq -> aten::lift_fresh -> aten::lift_fresh -> aten::to -> aten::to -> aten::empty -> aten::to -> aten::to -> aten::div -> aten::to_copy -> aten::to -> aten::contiguous -> aten::lift_fresh -> aten::local_scalar_dense -> aten::view -> aten::item -> aten::is_nonzero -> aten::to_copy -> aten::to -> aten::lt -> aten::to -> aten::lt -> aten::uniform -> aten::rand -> aten::item -> aten::random -> aten::empty -> aten::randint -> aten::item -> aten::random -> aten::empty -> aten::randint -> aten::sub_ -> aten::is_nonzero -> aten::view -> aten::fill_ -> aten::as_strided -> aten::any -> aten::lift_fresh. The conversion failed because a cyclic dependency was detected. Cyclic dependencies should not exist. The input and output traces must form a Directed Acyclic Graph (DAG). This is essential for simulation; otherwise, simulators cannot resolve the next dependency-free node and will hang. This indicates a bug in the conversion process. Please investigate or report this issue on GitHub.
ERROR [03/23/2025 09:28:44 AM] Cyclic dependency detected. The conversion failed because a cyclic dependency was detected. Cyclic dependencies should not exist. The input and output traces must form a Directed Acyclic Graph (DAG). This is essential for simulation; otherwise, simulators cannot resolve the next dependency-free node and will hang. This indicates a bug in the conversion process. Please investigate or report this issue on GitHub.
Traceback (most recent call last):
File "/home/marvell/alok/chakra/chakra_env/bin/chakra_converter", line 8, in
sys.exit(main())
File "/home/marvell/alok/chakra/chakra_env/lib/python3.10/site-packages/chakra/src/converter/converter.py", line 113, in main
args.func(args)
File "/home/marvell/alok/chakra/chakra_env/lib/python3.10/site-packages/chakra/src/converter/converter.py", line 36, in convert_pytorch
converter.convert(args.input, args.output, args.simulate)
File "/home/marvell/alok/chakra/chakra_env/lib/python3.10/site-packages/chakra/src/converter/pytorch_converter.py", line 57, in convert
self.identify_cyclic_dependencies(protobuf_node_map)
File "/home/marvell/alok/chakra/chakra_env/lib/python3.10/site-packages/chakra/src/converter/pytorch_converter.py", line 600, in identify_cyclic_dependencies
raise Exception(err_msg)
Exception: Cyclic dependency detected. The conversion failed because a cyclic dependency was detected. Cyclic dependencies should not exist. The input and output traces must form a Directed Acyclic Graph (DAG). This is essential for simulation; otherwise, simulators cannot resolve the next dependency-free node and will hang. This indicates a bug in the conversion process. Please investigate or report this issue on GitHub.

Note trace file is about 13GB.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/converter/pytorch_converter.py, especially convert and identify_cyclic_dependencies around the reported traceback lines. Reproduce the chakra_converter PyTorch command with the merged host/device trace and inspect how the 13GB input is converted. Done means the ResNet18 trace converts to a Protobuf file without the reported cyclic-dependency failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.