About chakra_trace_link

Hi, I am following your tutorial in wiki for Chakra ET generation with PyTorch ET and Kineto wiki . But when i try to merge the two trace from matmul, it raises the following error:

root@x08j03287:/workspace/astra-sim/extern/graph_frontend/chakra/example# chakra_trace_link --chakra-host-trace pytorch_et.json --chakra-device-trace kineto_trace.json --rank 0 --output-file chakra_host_device_trace.et
[2024-12-10 06:05:56,631] trace.py:328 [INFO]: /workspace/astra-sim/extern/graph_frontend/chakra/example
[2024-12-10 06:05:56,632] trace_file.py:56 [ERROR]: File /workspace/astra-sim/extern/graph_frontend/chakra/example/kineto_trace.json and file /workspace/astra-sim/extern/graph_frontend/chakra/example/pytorch_et.json has the same rank. Will use /workspace/astra-sim/extern/graph_frontend/chakra/example/pytorch_et.json as the path to rank: 0.
[2024-12-10 06:05:56,632] trace_file.py:94 [INFO]: Rank to trace file map:
{0: '/workspace/astra-sim/extern/graph_frontend/chakra/example/pytorch_et.json'}
[2024-12-10 06:05:56,632] trace.py:474 [INFO]: ranks=[0]
[2024-12-10 06:05:56,632] trace_parser.py:107 [WARNING]: Parsed /workspace/astra-sim/extern/graph_frontend/chakra/example/pytorch_et.json time = 0.00 seconds
[2024-12-10 06:05:56,633] trace_parser.py:430 [WARNING]: Parsed /workspace/astra-sim/extern/graph_frontend/chakra/example/pytorch_et.json backend=json in 0.00 seconds; current PID:58601
Traceback (most recent call last):
File "/usr/local/bin/chakra_trace_link", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/chakra/src/trace_link/trace_link.py", line 47, in main
linker.link(args.rank, args.chakra_host_trace, args.chakra_device_trace, args.output_file)
File "/usr/local/lib/python3.10/dist-packages/chakra/src/trace_link/trace_linker.py", line 70, in link
sync_deps = self.load_sync_dependencies(rank, chakra_device_trace)
File "/usr/local/lib/python3.10/dist-packages/chakra/src/trace_link/trace_linker.py", line 120, in load_sync_dependencies
trace_analysis = TraceAnalysis(trace_dir=trace_dir)
File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/trace_analysis.py", line 37, in init
self.t.load_traces(include_last_profiler_step)
File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/common/trace.py", line 352, in load_traces
self.parse_traces()
File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/common/trace.py", line 477, in parse_traces
self.parse_multiple_ranks(ranks, use_multiprocessing and len(ranks) > 1)
File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/common/trace.py", line 404, in parse_multiple_ranks
result = parse_trace_file(self.trace_files[rank])
File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/common/trace.py", line 228, in parse_trace_file
add_fwd_bwd_links(df)
File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/common/trace.py", line 244, in add_fwd_bwd_links
if df.cat.eq("fwdbwd").sum() == 0:
File "/usr/local/lib/python3.10/dist-packages/pandas/core/generic.py", line 6299, in getattr
return object.getattribute(self, name)
AttributeError: 'DataFrame' object has no attribute 'cat'. Did you mean: 'at'?

How to solve this problem?

Re-run the shown chakra_trace_link command, then inspect HolisticTraceAnalysis/hta/common/trace.py around parse_trace_file and add_fwd_bwd_links, along with src/trace_link/trace_linker.py around load_sync_dependencies. Determine why the parsed DataFrame lacks the expected cat attribute; done means the two trace files merge without the reported AttributeError.