googleapis / googleapis/python-aiplatform
Tensorboard uploading thread keeps program alive after exception kills main thread
- Langage dominant
- Python
- Étoiles
- 905
- Forks
- 465
- Merge moyen
- 1 j 13 h
- PR mergées (30 j)
- 44
Description
I spent a couple hundred dollars on vertex AI custom jobs that had errors but stayed running for a week because of this.
If it's bad to use a daemon thread here for some reason, can we at least add something to the docstring saying it's not a daemon thread and it will keep your program alive if it throws before calling end_upload_tb_log()?
Thanks!
#### Environment details
- OS type and version: Ubuntu 22.04
- Python version: `python --version`: Python 3.10.12
- pip version: `pip --version`: pip 23.2.1
- `google-cloud-aiplatform` version: `pip show google-cloud-aiplatform`: Version: 1.35.0
#### Steps to reproduce
1. Use aiplatform.start_upload_tb_log() with any code that throws an exception before end_upload_tb_log() is called.
2. Program will hang forever because the uploader uses a regular (non-daemon) thread to upload data.
#### Code example
```python
aiplatform.init(project=project,
credentials=cred,
location=location,
service_account=service_account)
# Start continuous monitoring
aiplatform.start_upload_tb_log(
tensorboard_id=tensorboard_id,
tensorboard_experiment_name=tensorboard_experiment_name,
logdir=logdir,
experiment_display_name=experiment_display_name,
run_name_prefix=run_name_prefix,
description=description,
)
# Run anything that throws an uncaught exception
raise Exception("oops")
# Stop continuous monitoring
aiplatform.end_upload_tb_log()
```
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.