pytorch / pytorch/kineto

Visualizing pytorch Emited NVTX with Tensorboard Profiler

Open
#787 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement plugin
Dominant language
C++
Stars
992
Forks
270
Avg merge
2d 11h
Merged PRs (30d)
12

Description

There are important Pytorch Specific markers generated during the training loop either using NVTX or other means from pytorch lightning.

It would be useful to show the time line view with execution time for different pytorch training sections of the code as shown in the attached image. The current trace view is quite detailed and goes down one-step further with detailed cuda kernels etc. It would be useful to have Pytorch Execution specific trace view that is easy to understand and intuit to find any synchronization or communication bottlenecks in the training loop and also compare different execution time for different executing region of the training/validation code (like forward, loss, data-loading etc).

Screenshot 2023-07-19 at 5 05 22 PM

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the current trace view and the attached timeline example, then investigate how PyTorch-specific NVTX or other training markers enter the profiler output. Define the scope and completion criteria for a PyTorch execution-focused view covering regions such as forward, loss, data loading, synchronization, and communication.

Written by the indexing model from the issue text.

Assessment

Tech stack
pytorch
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.