googleapis / googleapis/python-aiplatform
broken since 1.52.0: cannot call aiplatform.log_params after calling aiplatform.start_upload_tb_log
- Lingua principale
- Python
- Stelle
- 905
- Fork
- 465
- Merge medio
- 1g 13h
- PR unite (30g)
- 44
Descrizione
Calling aiplatform.start_upload_tb_log causes the aiplatform package to forget that it has started an Experiment run.
This bug was introduced in google-cloud-aiplatform==1.52.0. The code below with google-cloud-aiplatform==1.51.0 works. This commit is likely the source of the unwanted behavior:
https://github.com/googleapis/python-aiplatform/commit/339f8b667952c7302c36605842ba92fa1c7135b8
Here is a minimal example to reproduce, tested both locally on my laptop running MacOS Sequoia and in the cloud on a Workbench:
```
import google.cloud.aiplatform
def setup_experiment(
project: str,
location: str,
experiment_name: str,
experiment_version: str,
tensorboard_logs_path: str,
):
creds, _ = google.auth.default(
scopes=["https://www.googleapis.com/auth/cloud-platform"]
)
tensorboard = aiplatform.Tensorboard.create(
display_name=f"{experiment_name}-tensorboard",
project=project,
location=location,
credentials=creds,
)
aiplatform.init(
project=project,
location=location,
experiment=experiment_name,
experiment_tensorboard=tensorboard,
credentials=creds,
)
aiplatform.start_run(experiment_version)
aiplatform.start_upload_tb_log(
tensorboard_experiment_name=f"{experiment_name}-tb-{experiment_version}",
tensorboard_id=tensorboard.name,
logdir=tensorboard_logs_path,
description=experiment_version,
)
return tensorboard
project='[MY_PROJECT_ID]'
location='us-central1'
experiment_name = f'tensorboard-exp-{int(time())}'
experiment_run_name = f'run-{int(time())}'
tb_logs_dir = './logs'
os.makedirs(tb_logs_dir, exist_ok=True)
tensorboard = setup_experiment(
project=project,
location=location,
experiment_name=experiment_name,
experiment_version=experiment_run_name,
tensorboard_logs_path=tb_logs_dir
)
aiplatform.log_params({"param_1": 1, "param_2": "hi"})
```
The error received is:
```
File ~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:606, in _ExperimentTracker.log_params(self, params)
[591](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:591) def log_params(self, params: Dict[str, Union[float, int, str]]):
[592](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:592) """Log single or multiple parameters with specified key and value pairs.
[593](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:593)
[594](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:594) Parameters with the same key will be overwritten.
(...)
[603](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:603) Required. Parameter key/value pairs.
[604](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:604) """
--> [606](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:606) self._validate_experiment_and_run(method_name="log_params")
[607](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:607) # query the latest run execution resource before logging.
[608](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:608) self.experiment_run.log_params(params=params)
File ~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:788, in _ExperimentTracker._validate_experiment_and_run(self, method_name)
[783](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:783) raise ValueError(
[784](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:784) f"No experiment set. Make sure to call aiplatform.init(experiment='my-experiment') "
[785](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:785) f"before trying to {method_name}. "
[786](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:786) )
[787](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:787) if not self.experiment_run:
--> [788](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:788) raise ValueError(
[789](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:789) f"No run set. Make sure to call aiplatform.start_run('my-run') before trying to {method_name}. "
[790](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:790) )
ValueError: No run set. Make sure to call aiplatform.start_run('my-run') before trying to log_params.
```
If the change introduced in version 1.52 represents a new preferred way to interact with the SDK, the online documentation should be updated, and the pattern above prevented within the SDK.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.