tensorflow / tensorflow/tensorboard

Tensorboard callback write to mounted S3 path fails with GPU

Open
#5,676 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

core:summaries type:support
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Environment information (required)

Diagnostics
Diagnostics output 
--- check: autoidentify
INFO: diagnose_tensorboard.py version e43767ef2b648d0d5d57c00f38ccbd38390e38da

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=8, micro=9, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='28c43cd82c2f', release='3.10.0-1160.42.2.el7.x86_64', version='#1 SMP Tue Sep 7 14:49:57 UTC 2021', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==2.8.0
INFO: installed: tensorflow-gpu==2.8.0
INFO: installed: tf-estimator-nightly==2.8.0.dev2021122109
INFO: installed: tensorboard-data-server==0.6.1

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.8.0'

--- check: tensorflow_python_version
INFO: tensorflow.__version__: '2.8.0'
INFO: tensorflow.__git_version__: 'v2.8.0-rc1-32-g3f878cff5b6'

--- check: tensorboard_data_server_version
INFO: data server binary: '/usr/local/lib/python3.8/site-packages/tensorboard_data_server/bin/server'
INFO: failed to check binary version: Command '['/usr/local/lib/python3.8/site-packages/tensorboard_data_server/bin/server', '--version']' returned non-zero exit status 1.

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/usr/local/bin/tensorboard\n'

--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 32>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 32>
Loopback infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('0.0.0.0', 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 0, 0, 0))]

--- check: readable_fqdn
INFO: socket.getfqdn(): '28c43cd82c2f'

--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: .tensorboard-info directory does not exist

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/usr/local/lib/python3.8/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==1.0.0
astunparse==1.6.3
cachetools==5.0.0
certifi==2021.10.8
charset-normalizer==2.0.12
click==8.1.2
cloudpickle==2.0.0
cuda-python==11.5.0
cudf==22.4.0
cupy==9.5.0
cupy-cuda115==10.3.1
cycler==0.11.0
Cython==0.29.24
dask==2022.3.0
dask-cudf==22.4.0
distributed==2022.3.0
distro==1.7.0
dlpack==0.1
fastavro==1.4.5
fastrlock==0.8
flatbuffers==2.0
fsspec==2022.3.0
gast==0.5.3
google-auth==2.6.4
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
googleapis-common-protos==1.56.0
grpcio==1.44.0
h5py==3.6.0
HeapDict==1.0.1
idna==3.3
importlib-metadata==4.11.3
Jinja2==3.1.1
joblib==1.1.0
keras==2.8.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.2
libclang==13.0.0
llvmlite==0.37.0
locket==0.2.1
Markdown==3.3.6
MarkupSafe==2.1.1
matplotlib==3.2.1
msgpack==1.0.3
numba==0.54.1
numpy==1.20.0
nvidia-dali-cuda110==1.12.0
nvidia-dali-tf-plugin-cuda110==1.12.0
nvtx==0.2.3
oauthlib==3.2.0
opt-einsum==3.3.0
packaging==21.3
pandas==1.2.4
partd==1.2.0
pathlib2==2.3.7.post1
Pillow==9.0.1
pip==20.2.4
protobuf==3.20.0
psutil==5.9.0
pyarrow @ file:///opt/apache-arrow/python/dist/pyarrow-6.0.1-cp38-cp38-linux_x86_64.whl
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.8
python-dateutil==2.8.2
pytz==2022.1
PyYAML==5.3.1
requests==2.27.1
requests-oauthlib==1.3.1
rmm==21.12.0
rsa==4.8
scikit-build==0.14.1
scikit-learn==1.0.2
scipy==1.8.0
setuptools==61.3.1
six==1.16.0
sortedcontainers==2.4.0
tblib==1.7.0
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-gpu==2.8.0
tensorflow-io==0.24.0
tensorflow-io-gcs-filesystem==0.24.0
termcolor==1.1.0
tf-estimator-nightly==2.8.0.dev2021122109
threadpoolctl==3.1.0
toolz==0.11.2
tornado==6.1
typing-extensions==4.1.1
urllib3==1.26.9
verta==0.17.0
Werkzeug==2.1.1
wheel==0.37.1
wrapt==1.14.0
zict==2.1.0
zipp==3.8.0

Issue description

I'm running a TF training run, which writes TB logs to a rclone mounted S3 storage location (my own Minio installation).
The callback code for tensorboard is as follows:

logdir = os.path.join(TENSORBOARD_PATH, time.strftime("%Y%m%d-%H%M%S"))
file_writer = tf.summary.create_file_writer(logdir + "/metrics")

# log training metrics
parameter_summaries(writer=file_writer,
                            consts=[EPOCHS, BATCH_SIZE, PERIOD, INITAL_EPOCH, LEARNING_RATE, OPTIMIZER_TYPE],
                            names=["epochs", "batch_size", "period", "initial_epoch", "learning_rate",
                                   "optimizer_type"], step=0)

tensorboard = tf.keras.callbacks.TensorBoard(log_dir=logdir)
early_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, mode='min')
checkpointer = tf.keras.callbacks.ModelCheckpoint(monitor='val_loss', filepath=CHECKPOINTS_PATH,
                                                          save_weights_only=True, period=PERIOD, mode='min')
callbacks_list = [checkpointer, early_stopping, tensorboard]

This code is run in kubernetes, with a container that mounts the remote location during it's runtime.

rclone mount minio:${TENSORBOARD_BUCKET} /${TENSORBOARD_BUCKET} --daemon && \
model_training.py
umount /${TENSORBOARD_BUCKET}"]

When it is executed on the CPU, the code & logging run without problems

Epoch 19/20
1501/1501 [==============================] - 22s 15ms/step - loss: 0.0342 - accuracy: 0.9886 - precision: 1.0000 - recall: 0.9983 - f1_score: 0.9991 - val_loss: 0.0098 - val_accurac
y: 0.9972 - val_precision: 1.0000 - val_recall: 0.9993 - val_f1_score: 0.9996
Epoch 20/20
1501/1501 [==============================] - 22s 15ms/step - loss: 0.0323 - accuracy: 0.9899 - precision: 1.0000 - recall: 0.9984 - f1_score: 0.9992 - val_loss: 0.0101 - val_accurac
y: 0.9968 - val_precision: 1.0000 - val_recall: 0.9996 - val_f1_score: 0.9998
1501/1501 [==============================] - 8s 4ms/step - loss: 0.0095 - accuracy: 0.9972 - precision: 1.0000 - recall: 0.9998 - f1_score: 0.9999
313/313 [==============================] - 1s 4ms/step - loss: 0.0279 - accuracy: 0.9917 - precision: 1.0000 - recall: 0.9981 - f1_score: 0.9990
2022-04-19 13:28:02.400727: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
uploading part 1
upload complete (custom_modules)
uploading part 1
upload complete (model.pkl)
uploading part 1
upload complete (model_api.json)

But when I use the GPU, the tensorboard logging part fails:

2022-04-20 05:51:15,199 - INFO - model_training.py - Using distributed MirroredStrategy.
2022-04-20 05:51:15.200018: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the foll
owing CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-04-20 05:51:16.059450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9652 MB memory:  -> device: 0,
name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:1a:00.0, compute capability: 7.5
2022-04-20 05:51:16.060150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 9650 MB memory:  -> device: 1,
name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:68:00.0, compute capability: 7.5
WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of batches seen.
Epoch 1/5
2022-04-20 05:51:37.248551: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8302
2022-04-20 05:51:37.672889: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8302
1427/1501 [===========================>..] - ETA: 0s - loss: 0.5082 - accuracy: 0.8353 - precision: 0.952022-04-20 05:51:58.708918: W tensorflow/core/framework/op_kernel.cc:1745] OP
_REQUIRES failed at summary_kernels.cc:114 : FAILED_PRECONDITION: /tensorboard-bucket/mnist-images-e2e/mnist-e2e-11/20220420-055131/train/events.out.tfevents.1650433891.mnist-images
-e2e-run-w7s2q-3609785273.158.1.v2; Bad file descriptor
        Failed to flush 5 events to /tensorboard-bucket/mnist-images-e2e/mnist-e2e-11/20220420-055131/train/events.out.tfevents.1650433891.mnist-images-e2e-run-w7s2q-3609785273.158.
1.v2
        Could not flush events file.
    trained_model = trainer.train(train_params)
  File "/usr/local/lib/python3.8/site-packages/mnist_e2e/training/trainer.py", line 115, in train
    model.fit(train_ds.repeat().batch(BATCH_SIZE),
  File "/usr/local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 7186, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.FailedPreconditionError: /tensorboard-bucket/mnist-images-e2e/mnist-e2e-11/20220420-055131/train/events.out.tfevents.1650433891.mnist-images-
e2e-run-w7s2q-3609785273.158.1.v2; Bad file descriptor
        Failed to flush 5 events to /tensorboard-bucket/mnist-images-e2e/mnist-e2e-11/20220420-055131/train/events.out.tfevents.1650433891.mnist-images-e2e-run-w7s2q-3609785273.158.
1.v2
        Could not flush events file. [Op:FlushSummaryWriter]
1501/1501 [==============================] - ETA: 0s - loss: 0.4948 - accuracy: 0.8399 - precision: 0.9560 - recall: 0.8648 - f1_score: 0.9015

If I run the code on GPU and then copy the tensorboard logs afterwards, the workflow works normally. So the benefit I would like here is monitoring a tensorflow run as it is executed, via a remote S3 bucket mounted tensorboard. For a CPU run this works, but with a GPU I guess there are problems with files being written to the mounted disk too quickly or too many at a time? Are there any suggestions how this could be fixed, or is this an unfeasible approach?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the TensorBoard callback and tf.summary.create_file_writer path shown in the issue against an rclone-mounted Minio/S3 location, comparing CPU and GPU runs. Investigate the reported Bad file descriptor during summary flushing; done would be a confirmed cause and a validated fix or clear documentation of the supported limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
cloud, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.