Make continuous TensorBoard logging resilient to failed uploads

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

**Is your feature request related to a problem? Please describe.**

[aiplatform.start_upload_tb_log](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-upload-existing-logs#continuous-monitoring) currently is not resilient to any failures occurring when uploading. If an exception is thrown from an upload, then logging will stop and there is no way to restart it.

There is also no way to configure continuous logging to use `retry` arguments when calling an API to actually write to Vertex.

Specifically, [TensorBoardUploader.start_uploading](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/tensorboard/uploader.py#L332) has no exception handling or retry behaviour when calling `_upload_once` within the `_continue_uploading` while loop.

As well, [_TensorBoardTracker.start_upload_tb_log](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/tensorboard/uploader_tracker.py#L162) does not handle the possibility that `start_uploading` could fail. Therefore the logging thread can die if an exception occurs, but there is no way to check for this or restart the thread.

**Describe the solution you'd like**

Add mechanisms to `aiplatform.start_upload_tb_log` to allow continuous logging to continue if an upload fails, and/or to manually restart the uploading.

This could be done by providing a [Retry](https://googleapis.dev/python/google-api-core/latest/retry.html) to `start_upload_tb_log` which can then be passed down to all API calls.

To allow for manual restarting, `TensorBoardTracker.start_upload_tb_log` could store the running thread in a class field, and then provide a new function which will check if the thread is dead and restart it if so.

**Describe alternatives you've considered**

Another option would be to periodically call the [one-time logging](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-upload-existing-logs#one-time-logging) to upload any new events since the last upload, but this won't work because one-time logging will upload the whole TensorBoard each time.

A potential workaround I'm considering is constructing `_TensorBoardTracker` directly and periodically calling `_TensorBoardTracker._tensorboard_uploader._upload_once` to do one shot uploads containing only the new TensorBoard events since the last upload.

That way, if an upload fails, then the next triggered upload will still occur and will hopefully succeed.

**Additional context**

This has caused us to lose logging to Vertex during long training runs. The continuous logging encounters a connection issue partway through, and then after that nothing else is logged to Vertex.

We then need to manually manage the tensorboard outputs instead of being able to look at them in Vertex.

google/cloud/aiplatform/tensorboard/uploader.py の TensorBoardUploader.start_uploading、_continue_uploading、_upload_once 周辺を読み、続いて uploader_tracker.py の _TensorBoardTracker.start_upload_tb_log 周辺を調査してください。アップロードの失敗が logging スレッドにどのような影響を与えるかを追跡してください。実装では、継続的な logging が暗黙に停止するのではなく、再試行または復旧の動作を持つことを保証する必要があります。

索引モデルが issue の本文から書いたものです。