tensorflow / tensorflow/tensorboard

Conflict happened when using both tensorboard and Nsight System

Open
#5,289 2 comments 0 reactions 1 assignee View on GitHub

@yatbear is already working on this.

Since Sep 3, 2021.

stat:awaiting tensorflower type:support
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

System information

  • OS Platform Linux Ubuntu 18.04.5

  • TensorFlow installed from docker(tensorflow/tensorflow:2.4.3-gpu)

  • TensorFlow version :2.4.3-gpu (default)

  • Python version:3.6.9 (default)

  • CUDA version:11.0 (default)

  • cuDNN version:8.0.4 (default)

  • tensorboard version:2.6.0

  • GPU model and memory:NVIDIA A6000 / 48685MiB

Describe the current behavior

When using both Nsight System and tensorboard to profile my trainging , I found that nothing was catched in tensorboard . And when I used tensorboard only , it performed normal.

image-20210831193526750
56698dfb1de4d01d3f29f588fdfc99e

I'd like to know whether I can use Nsight System and tensorboard both. If I can ,what should I do; If I can't, tell me the reason please .

Thank you a lot.

Standalone code to reproduce the issue

The run scripts is like this. If I use it to start my traning , there will be conflicts.

nsys profile -t cuda,osrt,nvtx,cudnn,cublas \
       -o $1 \
       -f true \
       -w true \
       --sampling-period 2000000\
       python train.py

The core code of train.py is like this.

# ---preprocess dataset---
# ---get model and compile
# Add nvtx to nsys
import ctypes
_cuda_tools_ext = ctypes.CDLL("libnvToolsExt.so")
_cuda_tools_ext.nvtxRangePushA(ctypes.c_char_p("start train".encode('utf-8')))

tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir = "profile_log",
                                                      profile_batch = (1,391))
model.fit(train_db, 
        epochs=1, 
        validation_data = test_db,
        validation_freq=1,
        callbacks=[tensorboard_callback])

_cuda_tools_ext.nvtxRangePop()

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.