No RANK specified in the TRACE FILE
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 992
- Forks
- 270
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 12
Description
I tried to use the hta.trace_analysis.TraceAnalysis with trace files (json) create by the PyTorch profiler as recommended in the documentation.
with torch.profiler.profile(
schedule=torch.profiler.schedule(wait=1, warmup=1, active=3, repeat=1), #schedule the profiler to start after 1 step, warmup for 1 step, run for 3 steps and repeat 1 time
on_trace_ready= torch.profiler.tensorboard_trace_handler('./log/resnet18'), #save the trace to tensorboard by using a tracer object
record_shapes=True,
with_flops=True,
with_module_call_stack=True,
profile_memory=True,
with_stack=True
) as prof:
I get the error that no rank is specified in the trace file, I dont use distributed processes:
2024-08-08 14:03:38,854 - hta - trace_file.py:L61 - ERROR - If the trace file does not have the rank specified in it, then add the following snippet key to the json files to use HTA; "distributedInfo": {"rank": 0}. If there are multiple traces files, then each file should have a unique rank value.
2024-08-08 14:03:38,855 - hta - trace_file.py:L92 - WARNING - There is no item in the rank to trace file map.
2024-08-08 14:03:38,855 - hta - trace.py:L535 - INFO - ranks=[]
2024-08-08 14:03:38,856 - hta - trace.py:L541 - ERROR - The list of ranks to be parsed is empty.
What could be the problem? I saw that you work on the support for tracing distributed workloads is it related to this?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with hta.trace_analysis.TraceAnalysis and inspect the rank handling reported in hta/trace_file.py and hta/trace.py. Compare the PyTorch profiler JSON trace structure with the expected distributedInfo rank field, then verify that a single non-distributed trace can be parsed successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100