googleapis / googleapis/python-aiplatform

broken since 1.52.0: cannot call aiplatform.log_params after calling aiplatform.start_upload_tb_log

Offen
#5,061 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
api: vertex-ai
Vorherrschende Sprache
Python
Sterne
905
Forks
465
Ø Merge
1 T. 13 Std.
Gemergte PRs (30 T.)
44

Beschreibung

Calling aiplatform.start_upload_tb_log causes the aiplatform package to forget that it has started an Experiment run.

This bug was introduced in google-cloud-aiplatform==1.52.0. The code below with google-cloud-aiplatform==1.51.0 works. This commit is likely the source of the unwanted behavior:
https://github.com/googleapis/python-aiplatform/commit/339f8b667952c7302c36605842ba92fa1c7135b8

Here is a minimal example to reproduce, tested both locally on my laptop running MacOS Sequoia and in the cloud on a Workbench:

```
import google.cloud.aiplatform

def setup_experiment(
project: str,
location: str,
experiment_name: str,
experiment_version: str,
tensorboard_logs_path: str,
):

creds, _ = google.auth.default(
scopes=["https://www.googleapis.com/auth/cloud-platform"]
)

tensorboard = aiplatform.Tensorboard.create(
display_name=f"{experiment_name}-tensorboard",
project=project,
location=location,
credentials=creds,
)

aiplatform.init(
project=project,
location=location,
experiment=experiment_name,
experiment_tensorboard=tensorboard,
credentials=creds,
)
aiplatform.start_run(experiment_version)

aiplatform.start_upload_tb_log(
tensorboard_experiment_name=f"{experiment_name}-tb-{experiment_version}",
tensorboard_id=tensorboard.name,
logdir=tensorboard_logs_path,
description=experiment_version,
)

return tensorboard

project='[MY_PROJECT_ID]'
location='us-central1'
experiment_name = f'tensorboard-exp-{int(time())}'
experiment_run_name = f'run-{int(time())}'
tb_logs_dir = './logs'
os.makedirs(tb_logs_dir, exist_ok=True)

tensorboard = setup_experiment(
project=project,
location=location,
experiment_name=experiment_name,
experiment_version=experiment_run_name,
tensorboard_logs_path=tb_logs_dir
)

aiplatform.log_params({"param_1": 1, "param_2": "hi"})
```

The error received is:

```
File ~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:606, in _ExperimentTracker.log_params(self, params)
[591](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:591) def log_params(self, params: Dict[str, Union[float, int, str]]):
[592](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:592) """Log single or multiple parameters with specified key and value pairs.
[593](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:593)
[594](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:594) Parameters with the same key will be overwritten.
(...)
[603](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:603) Required. Parameter key/value pairs.
[604](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:604) """
--> [606](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:606) self._validate_experiment_and_run(method_name="log_params")
[607](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:607) # query the latest run execution resource before logging.
[608](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:608) self.experiment_run.log_params(params=params)

File ~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:788, in _ExperimentTracker._validate_experiment_and_run(self, method_name)
[783](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:783) raise ValueError(
[784](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:784) f"No experiment set. Make sure to call aiplatform.init(experiment='my-experiment') "
[785](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:785) f"before trying to {method_name}. "
[786](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:786) )
[787](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:787) if not self.experiment_run:
--> [788](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:788) raise ValueError(
[789](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:789) f"No run set. Make sure to call aiplatform.start_run('my-run') before trying to {method_name}. "
[790](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:790) )

ValueError: No run set. Make sure to call aiplatform.start_run('my-run') before trying to log_params.
```

If the change introduced in version 1.52 represents a new preferred way to interact with the SDK, the online documentation should be updated, and the pattern above prevented within the SDK.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

The traceback points to google/cloud/aiplatform/metadata/metadata.py, especially _ExperimentTracker.log_params and _validate_experiment_and_run; start by reproducing the shown sequence with versions 1.51.0 and 1.52.0. Done means the start_run followed by start_upload_tb_log still preserves the run so aiplatform.log_params succeeds, or the documentation clearly states the supported sequence.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
api, machine-learning
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
48/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.