Make continuous TensorBoard logging resilient to failed uploads

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

**Is your feature request related to a problem? Please describe.**

[aiplatform.start_upload_tb_log](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-upload-existing-logs#continuous-monitoring) currently is not resilient to any failures occurring when uploading. If an exception is thrown from an upload, then logging will stop and there is no way to restart it.

There is also no way to configure continuous logging to use `retry` arguments when calling an API to actually write to Vertex.

Specifically, [TensorBoardUploader.start_uploading](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/tensorboard/uploader.py#L332) has no exception handling or retry behaviour when calling `_upload_once` within the `_continue_uploading` while loop.

As well, [_TensorBoardTracker.start_upload_tb_log](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/tensorboard/uploader_tracker.py#L162) does not handle the possibility that `start_uploading` could fail. Therefore the logging thread can die if an exception occurs, but there is no way to check for this or restart the thread.

**Describe the solution you'd like**

Add mechanisms to `aiplatform.start_upload_tb_log` to allow continuous logging to continue if an upload fails, and/or to manually restart the uploading.

This could be done by providing a [Retry](https://googleapis.dev/python/google-api-core/latest/retry.html) to `start_upload_tb_log` which can then be passed down to all API calls.

To allow for manual restarting, `TensorBoardTracker.start_upload_tb_log` could store the running thread in a class field, and then provide a new function which will check if the thread is dead and restart it if so.

**Describe alternatives you've considered**

Another option would be to periodically call the [one-time logging](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-upload-existing-logs#one-time-logging) to upload any new events since the last upload, but this won't work because one-time logging will upload the whole TensorBoard each time.

A potential workaround I'm considering is constructing `_TensorBoardTracker` directly and periodically calling `_TensorBoardTracker._tensorboard_uploader._upload_once` to do one shot uploads containing only the new TensorBoard events since the last upload.

That way, if an upload fails, then the next triggered upload will still occur and will hopefully succeed.

**Additional context**

This has caused us to lose logging to Vertex during long training runs. The continuous logging encounters a connection issue partway through, and then after that nothing else is logged to Vertex.

We then need to manually manage the tensorboard outputs instead of being able to look at them in Vertex.

Read google/cloud/aiplatform/tensorboard/uploader.py around TensorBoardUploader.start_uploading, _continue_uploading, and _upload_once, then inspect uploader_tracker.py around _TensorBoardTracker.start_upload_tb_log. Trace how upload failures affect the logging thread; completion should ensure continuous logging has retry or recovery behavior instead of silently stopping.

Escrito por el modelo de indexación a partir del texto del issue.