googleapis / googleapis/python-aiplatform
batch_predict sync=False should create a job even if the shell exits
- Dominant language
- Python
- Stars
- 905
- Forks
- 465
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 44
Description
#### Environment details
- OS type and version: Google Composer (composer-2.1.7-airflow-2.4.3)
- Python version: 3.8.12
- pip version: 20.2.4
- `google-cloud-aiplatform` version: 1.16.1 (within composer-2.1.7-airflow-2.4.3)
#### Steps to reproduce
1. Run `aip.Model(model_name)..batch_predict(..., sync=False)` from your code
2. Exit the program/shell right after you issue this call.
3. The last lines you will see in the output before exiting:

4. If you go to the VertexAI console, you will see no indication that a batch predict request was even submitted/aborted. It's like it never happened.
5. Looking at the logs for VertexAI, we do see `The operation was cancelled`.

This sort of makes sense as the background execution/monitoring thread was terminated. However, it is surprising that there is no indication in the VertexAI console a batch prediction request was ever made.
#### Stack trace
N/A, silent failure
#### Expectation
I would expect `sync=False`, apart from giving me the ability to do something else in my program, to be truly async and have at least guaranteed the batch predict job is created/running.
A [similar issue](https://github.com/googleapis/python-aiplatform/issues/688) was opened and fixed for Jobs in general. In that case, `submit` was made available in addition to the regular `run`.
It would be great if sticking around was optional for `sync=False`. I do believe that is more in line with the semantics as well.
Contributor guide
Assessment
This issue has not been assessed yet.