googleapis / googleapis/python-aiplatform

Tensorboard uploading thread keeps program alive after exception kills main thread

Ouverte
#3,130 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub
api: vertex-ai
Langage dominant
Python
Étoiles
905
Forks
465
Merge moyen
1 j 13 h
PR mergées (30 j)
44

Description

I spent a couple hundred dollars on vertex AI custom jobs that had errors but stayed running for a week because of this.

If it's bad to use a daemon thread here for some reason, can we at least add something to the docstring saying it's not a daemon thread and it will keep your program alive if it throws before calling end_upload_tb_log()?

Thanks!

#### Environment details

- OS type and version: Ubuntu 22.04
- Python version: `python --version`: Python 3.10.12
- pip version: `pip --version`: pip 23.2.1
- `google-cloud-aiplatform` version: `pip show google-cloud-aiplatform`: Version: 1.35.0

#### Steps to reproduce

1. Use aiplatform.start_upload_tb_log() with any code that throws an exception before end_upload_tb_log() is called.
2. Program will hang forever because the uploader uses a regular (non-daemon) thread to upload data.

#### Code example

```python
aiplatform.init(project=project,
credentials=cred,
location=location,
service_account=service_account)

# Start continuous monitoring
aiplatform.start_upload_tb_log(
tensorboard_id=tensorboard_id,
tensorboard_experiment_name=tensorboard_experiment_name,
logdir=logdir,
experiment_display_name=experiment_display_name,
run_name_prefix=run_name_prefix,
description=description,
)

# Run anything that throws an uncaught exception
raise Exception("oops")

# Stop continuous monitoring
aiplatform.end_upload_tb_log()
```

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.