Make continuous TensorBoard logging resilient to failed uploads

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

**Is your feature request related to a problem? Please describe.**

[aiplatform.start_upload_tb_log](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-upload-existing-logs#continuous-monitoring) currently is not resilient to any failures occurring when uploading. If an exception is thrown from an upload, then logging will stop and there is no way to restart it.

There is also no way to configure continuous logging to use `retry` arguments when calling an API to actually write to Vertex.

Specifically, [TensorBoardUploader.start_uploading](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/tensorboard/uploader.py#L332) has no exception handling or retry behaviour when calling `_upload_once` within the `_continue_uploading` while loop.

As well, [_TensorBoardTracker.start_upload_tb_log](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/tensorboard/uploader_tracker.py#L162) does not handle the possibility that `start_uploading` could fail. Therefore the logging thread can die if an exception occurs, but there is no way to check for this or restart the thread.

**Describe the solution you'd like**

Add mechanisms to `aiplatform.start_upload_tb_log` to allow continuous logging to continue if an upload fails, and/or to manually restart the uploading.

This could be done by providing a [Retry](https://googleapis.dev/python/google-api-core/latest/retry.html) to `start_upload_tb_log` which can then be passed down to all API calls.

To allow for manual restarting, `TensorBoardTracker.start_upload_tb_log` could store the running thread in a class field, and then provide a new function which will check if the thread is dead and restart it if so.

**Describe alternatives you've considered**

Another option would be to periodically call the [one-time logging](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-upload-existing-logs#one-time-logging) to upload any new events since the last upload, but this won't work because one-time logging will upload the whole TensorBoard each time.

A potential workaround I'm considering is constructing `_TensorBoardTracker` directly and periodically calling `_TensorBoardTracker._tensorboard_uploader._upload_once` to do one shot uploads containing only the new TensorBoard events since the last upload.

That way, if an upload fails, then the next triggered upload will still occur and will hopefully succeed.

**Additional context**

This has caused us to lose logging to Vertex during long training runs. The continuous logging encounters a connection issue partway through, and then after that nothing else is logged to Vertex.

We then need to manually manage the tensorboard outputs instead of being able to look at them in Vertex.

Đọc google/cloud/aiplatform/tensorboard/uploader.py xung quanh TensorBoardUploader.start_uploading, _continue_uploading và _upload_once, sau đó kiểm tra uploader_tracker.py xung quanh _TensorBoardTracker.start_upload_tb_log. Theo dõi cách các lỗi upload ảnh hưởng đến logging thread; phần triển khai cần bảo đảm rằng logging liên tục có hành vi thử lại hoặc khôi phục thay vì âm thầm dừng lại.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.