googleapis / googleapis/python-aiplatform
Tensorboard uploading thread keeps program alive after exception kills main thread
- 主要言語
- Python
- スター
- 905
- フォーク
- 465
- 平均マージ
- 1日 13時間
- マージ済み PR(30日)
- 44
説明
I spent a couple hundred dollars on vertex AI custom jobs that had errors but stayed running for a week because of this.
If it's bad to use a daemon thread here for some reason, can we at least add something to the docstring saying it's not a daemon thread and it will keep your program alive if it throws before calling end_upload_tb_log()?
Thanks!
#### Environment details
- OS type and version: Ubuntu 22.04
- Python version: `python --version`: Python 3.10.12
- pip version: `pip --version`: pip 23.2.1
- `google-cloud-aiplatform` version: `pip show google-cloud-aiplatform`: Version: 1.35.0
#### Steps to reproduce
1. Use aiplatform.start_upload_tb_log() with any code that throws an exception before end_upload_tb_log() is called.
2. Program will hang forever because the uploader uses a regular (non-daemon) thread to upload data.
#### Code example
```python
aiplatform.init(project=project,
credentials=cred,
location=location,
service_account=service_account)
# Start continuous monitoring
aiplatform.start_upload_tb_log(
tensorboard_id=tensorboard_id,
tensorboard_experiment_name=tensorboard_experiment_name,
logdir=logdir,
experiment_display_name=experiment_display_name,
run_name_prefix=run_name_prefix,
description=description,
)
# Run anything that throws an uncaught exception
raise Exception("oops")
# Stop continuous monitoring
aiplatform.end_upload_tb_log()
```
コントリビューションガイド
評価
この issue はまだ評価されていません。