mlcommons / mlcommons/chakra

Traces without ProfilerStep no longer convertible since addition of HTA

Open
#162 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
196
Forks
84
PR merge metrics
No merged PRs in 30d

Description

Describe the Bug

I am collecting inference traces using the suggested calls to PyTorch profiler and am attempting to convert them using the latest code available for Chakra. Due to the addition of HTA, the trace linker now seems to rely on ProfilerStep annotations in the traces, otherwise the linking process will fail.

Steps to Reproduce

  • Collect traces using torch.profiler.profile, making use of profiler.start() and profiler.stop() but not profiler.step()
  • Attempt linking of traces using chakra_trace_link

Expected Behavior

  • A linked trace is created

Screenshots

Log output of chakra_trace_link:

WARNING:hta:Overall parsing of /home/.../PyCharmProjects/chakra/tests/data/new/device_trace.json in 1.24 seconds; current PID:206409
WARNING:hta:leaving parse_multiple_ranks duration=1.31 seconds
WARNING:hta:leaving parse_traces duration=1.31 seconds
WARNING:hta:ProfilerStep not found in the trace. The analysis result may not be accurate.
WARNING:hta:Trace does not contain CUDA Synchronization events so the results of analysis could be inaccurate.
WARNING:hta:Please see this PR to learn how to enable CUDA sync events https://github.com/pytorch/pytorch/pull/105187
ERROR:hta:Could not find annotation ProfilerStep in the trace.
Traceback (most recent call last):
  File "/home/.../PyCharmProjects/chakra/.venv/bin/chakra_trace_link", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/.../PyCharmProjects/chakra/.venv/lib/python3.13/site-packages/chakra/src/trace_link/trace_link.py", line 47, in main
    linker.link(args.rank, args.chakra_host_trace, args.chakra_device_trace, args.output_file)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.../PyCharmProjects/chakra/.venv/lib/python3.13/site-packages/chakra/src/trace_link/trace_linker.py", line 74, in link
    sync_deps = self.load_sync_dependencies(rank, chakra_device_trace)
  File "/home/.../PyCharmProjects/chakra/.venv/lib/python3.13/site-packages/chakra/src/trace_link/trace_linker.py", line 125, in load_sync_dependencies
    cp_graph, success = trace_analysis.critical_path_analysis(
    ^^^^^^^^^^^^^^^^^

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with torch.profiler.profile using profiler.start() and profiler.stop() without profiler.step(), then run chakra_trace_link on the resulting traces. Read trace_linker.py, especially link() and load_sync_dependencies(), along with the trace_analysis call shown in the traceback; done means a linked trace is created without requiring a ProfilerStep annotation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
cli, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.