googleapis / googleapis/python-aiplatform

broken since 1.52.0: cannot call aiplatform.log_params after calling aiplatform.start_upload_tb_log

Open
#5,061 0 comments 0 reactions 0 assignees View on GitHub
api: vertex-ai
Dominant language
Python
Stars
905
Forks
465
Avg merge
1d 13h
Merged PRs (30d)
44

Description

Calling aiplatform.start_upload_tb_log causes the aiplatform package to forget that it has started an Experiment run.

This bug was introduced in google-cloud-aiplatform==1.52.0. The code below with google-cloud-aiplatform==1.51.0 works. This commit is likely the source of the unwanted behavior:
https://github.com/googleapis/python-aiplatform/commit/339f8b667952c7302c36605842ba92fa1c7135b8

Here is a minimal example to reproduce, tested both locally on my laptop running MacOS Sequoia and in the cloud on a Workbench:

```
import google.cloud.aiplatform

def setup_experiment(
project: str,
location: str,
experiment_name: str,
experiment_version: str,
tensorboard_logs_path: str,
):

creds, _ = google.auth.default(
scopes=["https://www.googleapis.com/auth/cloud-platform"]
)

tensorboard = aiplatform.Tensorboard.create(
display_name=f"{experiment_name}-tensorboard",
project=project,
location=location,
credentials=creds,
)

aiplatform.init(
project=project,
location=location,
experiment=experiment_name,
experiment_tensorboard=tensorboard,
credentials=creds,
)
aiplatform.start_run(experiment_version)

aiplatform.start_upload_tb_log(
tensorboard_experiment_name=f"{experiment_name}-tb-{experiment_version}",
tensorboard_id=tensorboard.name,
logdir=tensorboard_logs_path,
description=experiment_version,
)

return tensorboard

project='[MY_PROJECT_ID]'
location='us-central1'
experiment_name = f'tensorboard-exp-{int(time())}'
experiment_run_name = f'run-{int(time())}'
tb_logs_dir = './logs'
os.makedirs(tb_logs_dir, exist_ok=True)

tensorboard = setup_experiment(
project=project,
location=location,
experiment_name=experiment_name,
experiment_version=experiment_run_name,
tensorboard_logs_path=tb_logs_dir
)

aiplatform.log_params({"param_1": 1, "param_2": "hi"})
```

The error received is:

```
File ~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:606, in _ExperimentTracker.log_params(self, params)
[591](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:591) def log_params(self, params: Dict[str, Union[float, int, str]]):
[592](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:592) """Log single or multiple parameters with specified key and value pairs.
[593](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:593)
[594](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:594) Parameters with the same key will be overwritten.
(...)
[603](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:603) Required. Parameter key/value pairs.
[604](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:604) """
--> [606](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:606) self._validate_experiment_and_run(method_name="log_params")
[607](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:607) # query the latest run execution resource before logging.
[608](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:608) self.experiment_run.log_params(params=params)

File ~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:788, in _ExperimentTracker._validate_experiment_and_run(self, method_name)
[783](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:783) raise ValueError(
[784](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:784) f"No experiment set. Make sure to call aiplatform.init(experiment='my-experiment') "
[785](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:785) f"before trying to {method_name}. "
[786](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:786) )
[787](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:787) if not self.experiment_run:
--> [788](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:788) raise ValueError(
[789](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:789) f"No run set. Make sure to call aiplatform.start_run('my-run') before trying to {method_name}. "
[790](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:790) )

ValueError: No run set. Make sure to call aiplatform.start_run('my-run') before trying to log_params.
```

If the change introduced in version 1.52 represents a new preferred way to interact with the SDK, the online documentation should be updated, and the pattern above prevented within the SDK.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.