googleapis / googleapis/python-aiplatform

Tensorboard uploading thread keeps program alive after exception kills main thread

Abierto
#3,130 1 comentario 0 reacciones 0 asignados Ver en GitHub
api: vertex-ai
Lenguaje dominante
Python
Estrellas
905
Forks
465
Merge medio
1 d 13 h
PR fusionados (30 d)
44

Descripción

I spent a couple hundred dollars on vertex AI custom jobs that had errors but stayed running for a week because of this.

If it's bad to use a daemon thread here for some reason, can we at least add something to the docstring saying it's not a daemon thread and it will keep your program alive if it throws before calling end_upload_tb_log()?

Thanks!

#### Environment details

- OS type and version: Ubuntu 22.04
- Python version: `python --version`: Python 3.10.12
- pip version: `pip --version`: pip 23.2.1
- `google-cloud-aiplatform` version: `pip show google-cloud-aiplatform`: Version: 1.35.0

#### Steps to reproduce

1. Use aiplatform.start_upload_tb_log() with any code that throws an exception before end_upload_tb_log() is called.
2. Program will hang forever because the uploader uses a regular (non-daemon) thread to upload data.

#### Code example

```python
aiplatform.init(project=project,
credentials=cred,
location=location,
service_account=service_account)

# Start continuous monitoring
aiplatform.start_upload_tb_log(
tensorboard_id=tensorboard_id,
tensorboard_experiment_name=tensorboard_experiment_name,
logdir=logdir,
experiment_display_name=experiment_display_name,
run_name_prefix=run_name_prefix,
description=description,
)

# Run anything that throws an uncaught exception
raise Exception("oops")

# Stop continuous monitoring
aiplatform.end_upload_tb_log()
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.