botocore.errorfactory.NoSuchKey when old TF Events got deleted

Consider Stack Overflow for getting support using TensorBoard—they have
a larger community with better searchability:

https://stackoverflow.com/questions/tagged/tensorboard

Do not use this template for for setup, installation, or configuration
issues. Instead, use the “installation problem” issue template:

https://github.com/tensorflow/tensorboard/issues/new?template=installation_problem.md

To report a problem with TensorBoard itself, please fill out the
remainder of this template.

Environment information (required)

Please run diagnose_tensorboard.py (link below) in the same
environment from which you normally run TensorFlow/TensorBoard, and
paste the output here:

tensorboard==2.9.1

https://raw.githubusercontent.com/tensorflow/tensorboard/master/tensorboard/tools/diagnose_tensorboard.py

For browser-related issues, please additionally specify:

• Browser type and version (e.g., Chrome 64.0.3282.140):
• Screenshot, if it’s a visual issue:

Issue description

Please describe the bug as clearly as possible. How can we reproduce the
problem without additional resources (including external data files and
proprietary Python modules)?

When use Tensorboard to read TFEvents from S3, the deleted TFEvents from the same logdir will trigger event_file_loader exceptions as following:

Exception in thread Reloader 15:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 239, in Worker
    accumulator.Reload()
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 183, in Reload
    for event in self._generator.Load():
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/backend/event_processing/directory_watcher.py", line 88, in Load
    for event in self._LoadInternal():
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/backend/event_processing/directory_watcher.py", line 118, in _LoadInternal
    for event in self._loader.Load():
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/backend/event_processing/event_file_loader.py", line 270, in Load
    for event in super(EventFileLoader, self).Load():
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/backend/event_processing/event_file_loader.py", line 244, in Load
    for record in super(LegacyEventFileLoader, self).Load():
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/backend/event_processing/event_file_loader.py", line 178, in Load
    yield next(self._iterator)
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/backend/event_processing/event_file_loader.py", line 109, in __next__
    self._reader.GetNext()
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 207, in GetNext
    header_str = self._read(8)
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 273, in _read
    new_data = self.file_handle.read(n)
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 727, in read
    (self.buff, self.continuation_token) = self.fs.read(
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 287, in read
    stream = s3.Object(bucket, path).get(**args)["Body"].read()
  File "/usr/local/lib/python3.10/dist-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.10/dist-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/dist-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

This exception will block any new event been processed and similar issue is: https://github.com/tensorflow/tensorboard/issues/2634

Start with tensorboard/backend/event_processing/event_file_loader.py and follow the traceback into directory_watcher.py and tensorboard/compat/tensorflow_stub/io/gfile.py. Reproduce the S3 case where an older TFEvent is deleted, then inspect how the resulting NoSuchKey exception stops event processing. Done means deleted event files no longer block newly available events from being processed.