Make continuous TensorBoard logging resilient to failed uploads

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

**Is your feature request related to a problem? Please describe.**

[aiplatform.start_upload_tb_log](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-upload-existing-logs#continuous-monitoring) currently is not resilient to any failures occurring when uploading. If an exception is thrown from an upload, then logging will stop and there is no way to restart it.

There is also no way to configure continuous logging to use `retry` arguments when calling an API to actually write to Vertex.

Specifically, [TensorBoardUploader.start_uploading](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/tensorboard/uploader.py#L332) has no exception handling or retry behaviour when calling `_upload_once` within the `_continue_uploading` while loop.

As well, [_TensorBoardTracker.start_upload_tb_log](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/tensorboard/uploader_tracker.py#L162) does not handle the possibility that `start_uploading` could fail. Therefore the logging thread can die if an exception occurs, but there is no way to check for this or restart the thread.

**Describe the solution you'd like**

Add mechanisms to `aiplatform.start_upload_tb_log` to allow continuous logging to continue if an upload fails, and/or to manually restart the uploading.

This could be done by providing a [Retry](https://googleapis.dev/python/google-api-core/latest/retry.html) to `start_upload_tb_log` which can then be passed down to all API calls.

To allow for manual restarting, `TensorBoardTracker.start_upload_tb_log` could store the running thread in a class field, and then provide a new function which will check if the thread is dead and restart it if so.

**Describe alternatives you've considered**

Another option would be to periodically call the [one-time logging](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-upload-existing-logs#one-time-logging) to upload any new events since the last upload, but this won't work because one-time logging will upload the whole TensorBoard each time.

A potential workaround I'm considering is constructing `_TensorBoardTracker` directly and periodically calling `_TensorBoardTracker._tensorboard_uploader._upload_once` to do one shot uploads containing only the new TensorBoard events since the last upload.

That way, if an upload fails, then the next triggered upload will still occur and will hopefully succeed.

**Additional context**

This has caused us to lose logging to Vertex during long training runs. The continuous logging encounters a connection issue partway through, and then after that nothing else is logged to Vertex.

We then need to manually manage the tensorboard outputs instead of being able to look at them in Vertex.

Lisez google/cloud/aiplatform/tensorboard/uploader.py autour de TensorBoardUploader.start_uploading, _continue_uploading et _upload_once, puis examinez uploader_tracker.py autour de _TensorBoardTracker.start_upload_tb_log. Suivez l’impact des échecs d’upload sur le thread de logging ; l’implémentation devrait garantir que le logging continu dispose d’un comportement de nouvelle tentative ou de récupération au lieu de s’arrêter silencieusement.

Rédigé par le modèle d'indexation à partir du texte de l'issue.