googleapis / googleapis/python-aiplatform
Tensorboard uploading thread keeps program alive after exception kills main thread
- 主要语言
- Python
- 星标
- 905
- 派生
- 465
- 平均合并
- 1 天 13 小时
- 30 天内合并 PR
- 44
描述
I spent a couple hundred dollars on vertex AI custom jobs that had errors but stayed running for a week because of this.
If it's bad to use a daemon thread here for some reason, can we at least add something to the docstring saying it's not a daemon thread and it will keep your program alive if it throws before calling end_upload_tb_log()?
Thanks!
#### Environment details
- OS type and version: Ubuntu 22.04
- Python version: `python --version`: Python 3.10.12
- pip version: `pip --version`: pip 23.2.1
- `google-cloud-aiplatform` version: `pip show google-cloud-aiplatform`: Version: 1.35.0
#### Steps to reproduce
1. Use aiplatform.start_upload_tb_log() with any code that throws an exception before end_upload_tb_log() is called.
2. Program will hang forever because the uploader uses a regular (non-daemon) thread to upload data.
#### Code example
```python
aiplatform.init(project=project,
credentials=cred,
location=location,
service_account=service_account)
# Start continuous monitoring
aiplatform.start_upload_tb_log(
tensorboard_id=tensorboard_id,
tensorboard_experiment_name=tensorboard_experiment_name,
logdir=logdir,
experiment_display_name=experiment_display_name,
run_name_prefix=run_name_prefix,
description=description,
)
# Run anything that throws an uncaught exception
raise Exception("oops")
# Stop continuous monitoring
aiplatform.end_upload_tb_log()
```
贡献指南
调研方向
从 aiplatform.start_upload_tb_log() 开始,跟踪 TensorBoard 上传流程使用的 uploader thread。在 end_upload_tb_log() 之前复现未捕获异常的情况,然后验证程序不再意外地保持运行,或者按照 issue 的要求,明确记录 non-daemon 行为。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- google-cloud, python
- 领域
- machine-learning
- Issue 类型
- 缺陷
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100