pytorch / pytorch/kineto

Module view dose not show device time

Open
#733 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug plugin
Dominant language
C++
Stars
992
Forks
270
Avg merge
2d 11h
Merged PRs (30d)
12

Description

Hi guys, I'm recently trying to use torch.profile for profiling of a large NLP model. However, I have encountered some problems and would like to get some advice:

  1. I found that only with_modules=True and with_stack=True are set at the same time, the tensorboard page will display the view of module. Otherwise, the module view option cannot be found in the browser, like:
    image

  2. In the view of module, there is no device time data, but I have added ProfilerActivity.CUDA in profile activities.

image

Here is an example of my code :

    with torch.profiler.profile(
        activities=[
            torch.profiler.ProfilerActivity.CPU,
            torch.profiler.ProfilerActivity.CUDA],
            schedule=torch.profiler.schedule(
                skip_first=5,
                warmup=1,
                wait=10,
                active=5),
            on_trace_ready=torch.profiler.tensorboard_trace_handler('./tensorboard'),
            with_modules = True,
            with_stack = True
        ) as prof:
          for i in range(steps):
             do training
             prof.step()

I'm not sure if this is a version related issue, here is my version info:

torch 1.13.1+cu117
torch-tb-profiler 0.4.1
torchaudio 0.13.1+cu117
torchvision 0.14.1+cu117

UPDATE: Today I upgraded torch version, but this problem still exists:

torch                   2.0.0.dev20230226+cu117
torch-tb-profiler       0.4.1
torchaudio              2.0.0.dev20230223+cu117
torchvision             0.15.0.dev20230226+cu117

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository file or test is named. Reproduce the provided torch.profiler example with the listed torch-tb-profiler versions, enabling CUDA activity, with_modules, and with_stack, then inspect the generated TensorBoard trace and module view. Done means the module view is available as expected and includes device-time data, or the limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.