googleapis / googleapis/python-aiplatform
broken since 1.52.0: cannot call aiplatform.log_params after calling aiplatform.start_upload_tb_log
- Dominant language
- Python
- Stars
- 905
- Forks
- 465
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 44
Description
Calling aiplatform.start_upload_tb_log causes the aiplatform package to forget that it has started an Experiment run.
This bug was introduced in google-cloud-aiplatform==1.52.0. The code below with google-cloud-aiplatform==1.51.0 works. This commit is likely the source of the unwanted behavior:
https://github.com/googleapis/python-aiplatform/commit/339f8b667952c7302c36605842ba92fa1c7135b8
Here is a minimal example to reproduce, tested both locally on my laptop running MacOS Sequoia and in the cloud on a Workbench:
```
import google.cloud.aiplatform
def setup_experiment(
project: str,
location: str,
experiment_name: str,
experiment_version: str,
tensorboard_logs_path: str,
):
creds, _ = google.auth.default(
scopes=["https://www.googleapis.com/auth/cloud-platform"]
)
tensorboard = aiplatform.Tensorboard.create(
display_name=f"{experiment_name}-tensorboard",
project=project,
location=location,
credentials=creds,
)
aiplatform.init(
project=project,
location=location,
experiment=experiment_name,
experiment_tensorboard=tensorboard,
credentials=creds,
)
aiplatform.start_run(experiment_version)
aiplatform.start_upload_tb_log(
tensorboard_experiment_name=f"{experiment_name}-tb-{experiment_version}",
tensorboard_id=tensorboard.name,
logdir=tensorboard_logs_path,
description=experiment_version,
)
return tensorboard
project='[MY_PROJECT_ID]'
location='us-central1'
experiment_name = f'tensorboard-exp-{int(time())}'
experiment_run_name = f'run-{int(time())}'
tb_logs_dir = './logs'
os.makedirs(tb_logs_dir, exist_ok=True)
tensorboard = setup_experiment(
project=project,
location=location,
experiment_name=experiment_name,
experiment_version=experiment_run_name,
tensorboard_logs_path=tb_logs_dir
)
aiplatform.log_params({"param_1": 1, "param_2": "hi"})
```
The error received is:
```
File ~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:606, in _ExperimentTracker.log_params(self, params)
[591](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:591) def log_params(self, params: Dict[str, Union[float, int, str]]):
[592](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:592) """Log single or multiple parameters with specified key and value pairs.
[593](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:593)
[594](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:594) Parameters with the same key will be overwritten.
(...)
[603](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:603) Required. Parameter key/value pairs.
[604](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:604) """
--> [606](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:606) self._validate_experiment_and_run(method_name="log_params")
[607](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:607) # query the latest run execution resource before logging.
[608](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:608) self.experiment_run.log_params(params=params)
File ~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:788, in _ExperimentTracker._validate_experiment_and_run(self, method_name)
[783](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:783) raise ValueError(
[784](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:784) f"No experiment set. Make sure to call aiplatform.init(experiment='my-experiment') "
[785](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:785) f"before trying to {method_name}. "
[786](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:786) )
[787](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:787) if not self.experiment_run:
--> [788](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:788) raise ValueError(
[789](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:789) f"No run set. Make sure to call aiplatform.start_run('my-run') before trying to {method_name}. "
[790](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:790) )
ValueError: No run set. Make sure to call aiplatform.start_run('my-run') before trying to log_params.
```
If the change introduced in version 1.52 represents a new preferred way to interact with the SDK, the online documentation should be updated, and the pattern above prevented within the SDK.
Contributor guide
Assessment
This issue has not been assessed yet.