googleapis / googleapis/python-aiplatform

broken since 1.52.0: cannot call aiplatform.log_params after calling aiplatform.start_upload_tb_log

Ouverte
#5,061 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
api: vertex-ai
Langage dominant
Python
Étoiles
905
Forks
465
Merge moyen
1 j 13 h
PR mergées (30 j)
44

Description

Calling aiplatform.start_upload_tb_log causes the aiplatform package to forget that it has started an Experiment run.

This bug was introduced in google-cloud-aiplatform==1.52.0. The code below with google-cloud-aiplatform==1.51.0 works. This commit is likely the source of the unwanted behavior:
https://github.com/googleapis/python-aiplatform/commit/339f8b667952c7302c36605842ba92fa1c7135b8

Here is a minimal example to reproduce, tested both locally on my laptop running MacOS Sequoia and in the cloud on a Workbench:

```
import google.cloud.aiplatform

def setup_experiment(
project: str,
location: str,
experiment_name: str,
experiment_version: str,
tensorboard_logs_path: str,
):

creds, _ = google.auth.default(
scopes=["https://www.googleapis.com/auth/cloud-platform"]
)

tensorboard = aiplatform.Tensorboard.create(
display_name=f"{experiment_name}-tensorboard",
project=project,
location=location,
credentials=creds,
)

aiplatform.init(
project=project,
location=location,
experiment=experiment_name,
experiment_tensorboard=tensorboard,
credentials=creds,
)
aiplatform.start_run(experiment_version)

aiplatform.start_upload_tb_log(
tensorboard_experiment_name=f"{experiment_name}-tb-{experiment_version}",
tensorboard_id=tensorboard.name,
logdir=tensorboard_logs_path,
description=experiment_version,
)

return tensorboard

project='[MY_PROJECT_ID]'
location='us-central1'
experiment_name = f'tensorboard-exp-{int(time())}'
experiment_run_name = f'run-{int(time())}'
tb_logs_dir = './logs'
os.makedirs(tb_logs_dir, exist_ok=True)

tensorboard = setup_experiment(
project=project,
location=location,
experiment_name=experiment_name,
experiment_version=experiment_run_name,
tensorboard_logs_path=tb_logs_dir
)

aiplatform.log_params({"param_1": 1, "param_2": "hi"})
```

The error received is:

```
File ~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:606, in _ExperimentTracker.log_params(self, params)
[591](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:591) def log_params(self, params: Dict[str, Union[float, int, str]]):
[592](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:592) """Log single or multiple parameters with specified key and value pairs.
[593](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:593)
[594](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:594) Parameters with the same key will be overwritten.
(...)
[603](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:603) Required. Parameter key/value pairs.
[604](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:604) """
--> [606](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:606) self._validate_experiment_and_run(method_name="log_params")
[607](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:607) # query the latest run execution resource before logging.
[608](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:608) self.experiment_run.log_params(params=params)

File ~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:788, in _ExperimentTracker._validate_experiment_and_run(self, method_name)
[783](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:783) raise ValueError(
[784](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:784) f"No experiment set. Make sure to call aiplatform.init(experiment='my-experiment') "
[785](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:785) f"before trying to {method_name}. "
[786](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:786) )
[787](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:787) if not self.experiment_run:
--> [788](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:788) raise ValueError(
[789](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:789) f"No run set. Make sure to call aiplatform.start_run('my-run') before trying to {method_name}. "
[790](https://file+.vscode-resource.vscode-cdn.net/[PATH_TO_MY_NOTEBOOK]/~/.pyenv/versions/nih/lib/python3.11/site-packages/google/cloud/aiplatform/metadata/metadata.py:790) )

ValueError: No run set. Make sure to call aiplatform.start_run('my-run') before trying to log_params.
```

If the change introduced in version 1.52 represents a new preferred way to interact with the SDK, the online documentation should be updated, and the pattern above prevented within the SDK.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

La traceback pointe vers google/cloud/aiplatform/metadata/metadata.py, en particulier vers _ExperimentTracker.log_params et _validate_experiment_and_run ; commencez par reproduire la séquence indiquée avec les versions 1.51.0 et 1.52.0. C’est terminé lorsque start_run suivi de start_upload_tb_log préserve toujours le run, de sorte que aiplatform.log_params réussisse, ou lorsque la documentation indique clairement la séquence prise en charge.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
api, machine-learning
Type d'issue
Bug
Difficulté
3/5
Temps estimé
1-2 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
48/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.