googleapis / googleapis/python-aiplatform

Tensorboard uploading thread keeps program alive after exception kills main thread

未关闭
#3,130 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
api: vertex-ai
主要语言
Python
星标
905
派生
465
平均合并
1 天 13 小时
30 天内合并 PR
44

描述

I spent a couple hundred dollars on vertex AI custom jobs that had errors but stayed running for a week because of this.

If it's bad to use a daemon thread here for some reason, can we at least add something to the docstring saying it's not a daemon thread and it will keep your program alive if it throws before calling end_upload_tb_log()?

Thanks!

#### Environment details

- OS type and version: Ubuntu 22.04
- Python version: `python --version`: Python 3.10.12
- pip version: `pip --version`: pip 23.2.1
- `google-cloud-aiplatform` version: `pip show google-cloud-aiplatform`: Version: 1.35.0

#### Steps to reproduce

1. Use aiplatform.start_upload_tb_log() with any code that throws an exception before end_upload_tb_log() is called.
2. Program will hang forever because the uploader uses a regular (non-daemon) thread to upload data.

#### Code example

```python
aiplatform.init(project=project,
credentials=cred,
location=location,
service_account=service_account)

# Start continuous monitoring
aiplatform.start_upload_tb_log(
tensorboard_id=tensorboard_id,
tensorboard_experiment_name=tensorboard_experiment_name,
logdir=logdir,
experiment_display_name=experiment_display_name,
run_name_prefix=run_name_prefix,
description=description,
)

# Run anything that throws an uncaught exception
raise Exception("oops")

# Stop continuous monitoring
aiplatform.end_upload_tb_log()
```

贡献指南

打开贡献指南

调研方向

从 aiplatform.start_upload_tb_log() 开始,跟踪 TensorBoard 上传流程使用的 uploader thread。在 end_upload_tb_log() 之前复现未捕获异常的情况,然后验证程序不再意外地保持运行,或者按照 issue 的要求,明确记录 non-daemon 行为。

由索引模型根据 Issue 内容生成。

评估

技术栈
google-cloud, python
领域
machine-learning
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。