pytorch / pytorch/kineto

No RANK specified in the TRACE FILE

Open
#974 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
992
Forks
270
Avg merge
2d 11h
Merged PRs (30d)
12

Description

I tried to use the hta.trace_analysis.TraceAnalysis with trace files (json) create by the PyTorch profiler as recommended in the documentation.

with torch.profiler.profile( schedule=torch.profiler.schedule(wait=1, warmup=1, active=3, repeat=1), #schedule the profiler to start after 1 step, warmup for 1 step, run for 3 steps and repeat 1 time on_trace_ready= torch.profiler.tensorboard_trace_handler('./log/resnet18'), #save the trace to tensorboard by using a tracer object record_shapes=True, with_flops=True, with_module_call_stack=True, profile_memory=True, with_stack=True ) as prof:

I get the error that no rank is specified in the trace file, I dont use distributed processes:

2024-08-08 14:03:38,854 - hta - trace_file.py:L61 - ERROR - If the trace file does not have the rank specified in it, then add the following snippet key to the json files to use HTA; "distributedInfo": {"rank": 0}. If there are multiple traces files, then each file should have a unique rank value. 2024-08-08 14:03:38,855 - hta - trace_file.py:L92 - WARNING - There is no item in the rank to trace file map. 2024-08-08 14:03:38,855 - hta - trace.py:L535 - INFO - ranks=[] 2024-08-08 14:03:38,856 - hta - trace.py:L541 - ERROR - The list of ranks to be parsed is empty.

What could be the problem? I saw that you work on the support for tracing distributed workloads is it related to this?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with hta.trace_analysis.TraceAnalysis and inspect the rank handling reported in hta/trace_file.py and hta/trace.py. Compare the PyTorch profiler JSON trace structure with the expected distributedInfo rank field, then verify that a single non-distributed trace can be parsed successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.