GPU traces fail when using PyTorch lightning due to square braces in traceName
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 992
- Forks
- 270
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 12
Description
PyTorch Lightning saves traces in the format fit-profiler-[Strategy]SingleDeviceStrategy-ts.pt.trace.json, and thus the traceName key contains a square brace ]. Since the traceName key is the final key in the JSON file, it fails to load in the TensorBoard viewer when GPU ops are present due to this line, which cuts off the string and creates invalid JSON:
https://github.com/pytorch/kineto/blob/8466a8b111b36dc725e6855d52a0b133d925a8e0/tb_plugin/torch_tb_profiler/run.py#L137
This results in the error Uncaught (in promise) SyntaxError: JSON.parse, ... in the web browser.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at tb_plugin/torch_tb_profiler/run.py line 137, where the traceName string is cut off. Reproduce the issue with a PyTorch Lightning trace whose name contains a square bracket and verify that the TensorBoard viewer can parse the resulting JSON when GPU operations are present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- observability, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100