tensorflow / tensorflow/tensorboard
Conflict happened when using both tensorboard and Nsight System
@yatbear is already working on this.
Since Sep 3, 2021.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
System information
-
OS Platform Linux Ubuntu 18.04.5
-
TensorFlow installed from docker(tensorflow/tensorflow:2.4.3-gpu)
-
TensorFlow version :2.4.3-gpu (default)
-
Python version:3.6.9 (default)
-
CUDA version:11.0 (default)
-
cuDNN version:8.0.4 (default)
-
tensorboard version:2.6.0
-
GPU model and memory:NVIDIA A6000 / 48685MiB
Describe the current behavior
When using both Nsight System and tensorboard to profile my trainging , I found that nothing was catched in tensorboard . And when I used tensorboard only , it performed normal.


I'd like to know whether I can use Nsight System and tensorboard both. If I can ,what should I do; If I can't, tell me the reason please .
Thank you a lot.
Standalone code to reproduce the issue
The run scripts is like this. If I use it to start my traning , there will be conflicts.
nsys profile -t cuda,osrt,nvtx,cudnn,cublas \
-o $1 \
-f true \
-w true \
--sampling-period 2000000\
python train.py
The core code of train.py is like this.
# ---preprocess dataset---
# ---get model and compile
# Add nvtx to nsys
import ctypes
_cuda_tools_ext = ctypes.CDLL("libnvToolsExt.so")
_cuda_tools_ext.nvtxRangePushA(ctypes.c_char_p("start train".encode('utf-8')))
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir = "profile_log",
profile_batch = (1,391))
model.fit(train_db,
epochs=1,
validation_data = test_db,
validation_freq=1,
callbacks=[tensorboard_callback])
_cuda_tools_ext.nvtxRangePop()
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.