Using Tensorboard on model with multiple inputs and embeddings_data returns "ValueError: No variables to save"

Environment information (required)

Please run diagnose_tensorboard.py (link below) in the same
environment from which you normally run TensorFlow/TensorBoard, and
paste the output here:

Diagnostics

--- check: autoidentify
INFO: diagnose_tensorboard.py version 724b56cee52e7d8eb89bbeec1f0d5ce3e38c9682

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
INFO: os.name: nt
INFO: os.uname(): N/A
INFO: sys.getwindowsversion(): sys.getwindowsversion(major=10, minor=0, build=18363, platform=2, service_pack='')

--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==1.14.0
INFO: installed: tensorflow==1.14.0
INFO: installed: tensorflow-estimator==1.14.0

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '1.14.0'

--- check: tensorflow_python_version
INFO: tensorflow.__version__: '1.14.0'
INFO: tensorflow.__git_version__: 'unknown'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'C:\\Users\\me\\.conda\\envs\\tf_gpu\\Scripts\\tensorboard.exe\r\n'

--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 1024>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 1024>
Loopback infos: [(<AddressFamily.AF_INET6: 23>, <SocketKind.SOCK_STREAM: 1>, 0, '', ('::1', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 0, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET6: 23>, <SocketKind.SOCK_STREAM: 1>, 0, '', ('::', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 0, '', ('0.0.0.0', 0))]

--- check: readable_fqdn
INFO: socket.getfqdn(): 'IT066379.wks.bris.ac.uk'

--- check: stat_tensorboardinfo
INFO: directory: C:\Users\me\AppData\Local\Temp\.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=14636698788984755, st_dev=4199001745, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1603112547, st_mtime=1603112547, st_ctime=1603112547)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['C:\\Users\\me\\.conda\\envs\\tf_gpu\\lib\\site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.8.0
astor==0.8.0
backcall==0.1.0
certifi==2020.6.20
colorama==0.4.1
cycler==0.10.0
decorator==4.4.0
gast==0.3.2
grpcio==1.16.1
h5py==2.8.0
imageio==2.6.1
imutils==0.5.3
ipython==7.8.0
ipython-genutils==0.2.0
jedi==0.15.1
joblib==0.13.2
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
matplotlib==3.1.1
memory-profiler==0.55.0
mkl-fft==1.0.14
mkl-random==1.1.0
mkl-service==2.3.0
numpy==1.16.5
olefile==0.46
pandas==0.25.2
parso==0.5.1
pickleshare==0.7.5
Pillow==6.2.0
pip==19.3.1
prompt-toolkit==2.0.10
protobuf==3.9.2
psutil==5.6.7
pydot==1.4.1
Pygments==2.4.2
pymongo==3.9.0
pyparsing==2.4.2
pyreadline==2.1
python-dateutil==2.8.0
pytz==2019.3
PyYAML==5.1.2
scikit-learn==0.21.3
scikit-video==1.1.11
scipy==1.3.1
setuptools==41.4.0
six==1.12.0
# Editable install with no version control (sphere==0.1)
-e c:\users\me\dropbox\
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
termcolor==1.1.0
tornado==6.0.3
tqdm==4.36.1
traitlets==4.3.3
wcwidth==0.1.7
Werkzeug==0.16.0
wheel==0.33.6
wincertstore==0.2
wrapt==1.11.2

Next steps

No action items identified. Please copy ALL of the above output,
including the lines containing only backticks, into your GitHub issue
or comment. Be sure to redact any sensitive information.

Issue description

Here is an example model with a variable number of inputs:

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense, add
from tensorflow.keras.callbacks import TensorBoard
import numpy as np

num_inputs = 4
inputs = []
for i in range(num_inputs):
    inputs.append(Input(shape=(32,), name='input_'+str(i)))

merged = add([Dense(16, name='dense_'+str(i))(inpt) for i,inpt in enumerate(inputs)],
             name='merge')
o = Dense(64, name='dense_o')(merged)

model = Model(inputs=inputs, outputs=o)
model.compile(optimizer='sgd', loss='mean_squared_error')

batch_size = 100
x_train = [np.random.randn(batch_size, 32) for i in range(num_inputs)]
y_train = np.random.randn(batch_size, 64)

x_val = [np.random.randn(batch_size, 32) for i in range(num_inputs)]
y_val = np.random.randn(batch_size, 64)
model.fit(x=x_train, y=y_train,
          validation_data=[x_val, y_val],
          callbacks=[TensorBoard(histogram_freq=5, embeddings_freq=5, embeddings_data=x_val)], epochs=100)

When embeddings_freq is specified, with embeddings_data, the code returns this error:

Traceback (most recent call last):
  File "C:\Users\~\.conda\envs\tf_gpu\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-409f940a576d>", line 1, in <module>
    runfile('~/delme.py', wdir='~')
  File "C:\Program Files\JetBrains\PyCharm 2019.2.3\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2019.2.3\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "~/delme.py", line 26, in <module>
    callbacks=[TensorBoard(histogram_freq=5, embeddings_freq=5, embeddings_data=x_val)], epochs=100)
  File "C:\Users\~\.conda\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\engine\training.py", line 780, in fit
    steps_name='steps_per_epoch')
  File "C:\Users\~\.conda\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py", line 213, in model_iteration
    mode=mode)
  File "C:\Users\~\.conda\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\callbacks.py", line 105, in configure_callbacks
    callback_list.set_model(callback_model)
  File "C:\Users\~\.conda\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\callbacks.py", line 231, in set_model
    callback.set_model(model)
  File "C:\Users\~\.conda\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\callbacks_v1.py", line 280, in set_model
    self.saver = saver.Saver(list(embeddings_vars.values()))
  File "C:\Users\~\.conda\envs\tf_gpu\lib\site-packages\tensorflow\python\training\saver.py", line 825, in __init__
    self.build()
  File "C:\Users\~\.conda\envs\tf_gpu\lib\site-packages\tensorflow\python\training\saver.py", line 837, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "C:\Users\~\.conda\envs\tf_gpu\lib\site-packages\tensorflow\python\training\saver.py", line 862, in _build
    raise ValueError("No variables to save")
ValueError: No variables to save

When embedding_freq is removed, the code runs smoothly:

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense, add
from tensorflow.keras.callbacks import TensorBoard
import numpy as np

num_inputs = 4
inputs = []
for i in range(num_inputs):
    inputs.append(Input(shape=(32,), name='input_'+str(i)))

merged = add([Dense(16, name='dense_'+str(i))(inpt) for i,inpt in enumerate(inputs)],
             name='merge')
o = Dense(64, name='dense_o')(merged)

model = Model(inputs=inputs, outputs=o)
model.compile(optimizer='sgd', loss='mean_squared_error')

batch_size = 100
x_train = [np.random.randn(batch_size, 32) for i in range(num_inputs)]
y_train = np.random.randn(batch_size, 64)

x_val = [np.random.randn(batch_size, 32) for i in range(num_inputs)]
y_val = np.random.randn(batch_size, 64)
model.fit(x=x_train, y=y_train,
          validation_data=[x_val, y_val],
          callbacks=[TensorBoard(histogram_freq=5)], epochs=100)

Please describe the bug as clearly as possible. How can we reproduce the
problem without additional resources (including external data files and
proprietary Python modules)?