googleapis / googleapis/python-aiplatform

ModelServiceClient.import_model_evaluation does not work as expected

オープン
#5,570 コメント 1 件 リアクション 1 件 担当者 0 名 GitHub で見る
api: vertex-ai
主要言語
Python
スター
905
フォーク
465
平均マージ
1日 13時間
マージ済み PR(30日)
44

説明

I wanted to upload a model evaluation following the gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml and the steps described in https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb. The code snippet in the tutorial works fine, but I want to add a "confusionMatrix", as defined by the metric schema. The same code snippet now doesn't work anymore, and the import_model_evaluation returns a "400 Invalid Metrics Format" error.

To find out the issue, I have tried to print out a model evaluation created by a automl tabular job and upload said evaluation to a custom model, but this also results in the same 400 error. The automl evaluation is reported to follow the metric schema, as it is returned as its metricsSchemaUri. It seems to me the reported evaluation results do not follow the metric schema, as the "rows" contain arrays of arrays, instead of the expected arrays of objects of arrays required by the schema yaml. I have attached a log file with the full exported evaluation metric used for the test [automl.log](https://github.com/user-attachments/files/21312310/automl.log)

I have also tested uploading a reduced form of the evaluation metrics dictionary in two formats:

```
{
"confusionMatrix": {
"annotationSpecs": [
{"displayName": "Husband"},
{"displayName": "Not-in-family"},
{"displayName": "Own-child"},
{"displayName": "Wife"},
{"displayName": "Unmarried"},
{"displayName": "Other-relative"},
],
"rows": [
{"row": [1614, 0, 0, 0, 0, 0]},
{"row": [2, 805, 1, 124, 96, 1]},
{"row": [11, 52, 2, 37, 12, 4]},
{"row": [5, 188, 2, 359, 30, 6]},
{"row": [0, 202, 1, 28, 187, 0]},
{"row": [0, 0, 0, 0, 0, 186]},
],
},
"auPrc": 0.8,
"auRoc": 0.8,
"logLoss": 0.5,
}
```

and

```
{
"confusionMatrix": {
"annotationSpecs": [
{"displayName": "Husband"},
{"displayName": "Not-in-family"},
{"displayName": "Own-child"},
{"displayName": "Wife"},
{"displayName": "Unmarried"},
{"displayName": "Other-relative"},
],
"rows": [
[1614, 0, 0, 0, 0, 0],
[2, 805, 1, 124, 96, 1],
[11, 52, 2, 37, 12, 4],
[5, 188, 2, 359, 30, 6],
[0, 202, 1, 28, 187, 0],
[0, 0, 0, 0, 0, 186],
],
},
"auPrc": 0.8,
"auRoc": 0.8,
"logLoss": 0.5,
}
```

Both do not work and will return a 400.

#### Environment details

- OS type and version: MacOS Sequoia 15.5
- Python version: 3.12.11
- pip version: 25.1.1
- `google-cloud-aiplatform` version:
Name: google-cloud-aiplatform
Version: 1.92.0
Summary: Vertex AI API client library
Home-page: https://github.com/googleapis/python-aiplatform
Author: Google LLC
Author-email: googleapis-packages@google.com
License: Apache 2.0
Location: -
Requires: docstring-parser, google-api-core, google-auth, google-cloud-bigquery, google-cloud-resource-manager, google-cloud-storage, google-genai, packaging, proto-plus, protobuf, pydantic, shapely, typing-extensions
Required-by:

#### Steps to reproduce

1. Follow https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb#scrollTo=852be5e5f964
2. Instead of just uploading the small evaluation {"logLoss": ...,"auRoc": ...} use the provided metric dicts with confusionMatrix
4. Observe the metric invalid error

#### Stack trace
```
---------------------------------------------------------------------------
_InactiveRpcError Traceback (most recent call last)
File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:76, in _wrap_unary_errors..error_remapped_callable(*args, **kwargs)
75 try:
---> [76](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:76) return callable_(*args, **kwargs)
77 except grpc.RpcError as exc:

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:277, in _UnaryUnaryMultiCallable.__call__(self, request, timeout, metadata, credentials, wait_for_ready, compression)
268 def __call__(
269 self,
270 request: Any,
(...) 275 compression: Optional[grpc.Compression] = None,
276 ) -> Any:
--> [277](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:277) response, ignored_call = self._with_call(
278 request,
279 timeout=timeout,
280 metadata=metadata,
281 credentials=credentials,
282 wait_for_ready=wait_for_ready,
283 compression=compression,
284 )
285 return response

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:332, in _UnaryUnaryMultiCallable._with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
329 call = self._interceptor.intercept_unary_unary(
330 continuation, client_call_details, request
331 )
--> [332](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:332) return call.result(), call

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:440, in _InactiveRpcError.result(self, timeout)
439 """See grpc.Future.result."""
--> [440](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:440) raise self

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:315, in _UnaryUnaryMultiCallable._with_call..continuation(new_details, request)
314 try:
--> [315](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:315) response, call = self._thunk(new_method).with_call(
316 request,
317 timeout=new_timeout,
318 metadata=new_metadata,
319 credentials=new_credentials,
320 wait_for_ready=new_wait_for_ready,
321 compression=new_compression,
322 )
323 return _UnaryOutcome(response, call)

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1198, in _UnaryUnaryMultiCallable.with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
1192 (
1193 state,
1194 call,
1195 ) = self._blocking(
1196 request, timeout, metadata, credentials, wait_for_ready, compression
1197 )
-> [1198](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1198) return _end_unary_response_blocking(state, call, True, None)

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1006, in _end_unary_response_blocking(state, call, with_call, deadline)
1005 else:
-> [1006](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1006) raise _InactiveRpcError(state)

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Invalid Metrics format."
debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B2a00:1450:4001:80f::200a%5D:443 {created_time:"2025-07-18T09:18:03.606562+02:00", grpc_status:3, grpc_message:"Invalid Metrics format."}"
>

The above exception was the direct cause of the following exception:

InvalidArgument Traceback (most recent call last)
Cell In[64], [line 14](vscode-notebook-cell:?execution_count=64&line=14)
6 model_eval = ModelEvaluation(
7 display_name=f"{EXPERIMENT_NAME}-eval",
8 metrics_schema_uri="gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
9 metrics=fixed_dict,
10 )
11 client = ModelServiceClient(
12 client_options={"api_endpoint": f"{LOCATION}-aiplatform.googleapis.com"}
13 )
---> [14](vscode-notebook-cell:?execution_count=64&line=14) client.import_model_evaluation(parent=model.resource_name, model_evaluation=model_eval)

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/model_service/client.py:2592, in ModelServiceClient.import_model_evaluation(self, request, parent, model_evaluation, retry, timeout, metadata)
2589 self._validate_universe_domain()
2591 # Send the request.
-> [2592](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/model_service/client.py:2592) response = rpc(
2593 request,
2594 retry=retry,
2595 timeout=timeout,
2596 metadata=metadata,
2597 )
2599 # Done; return the response.
2600 return response

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py:131, in _GapicCallable.__call__(self, timeout, retry, compression, *args, **kwargs)
128 if self._compression is not None:
129 kwargs["compression"] = compression
--> [131](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py:131) return wrapped_func(*args, **kwargs)

File ~/Projekte/Sandbox/sandbox/autogluon/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:78, in _wrap_unary_errors..error_remapped_callable(*args, **kwargs)
76 return callable_(*args, **kwargs)
77 except grpc.RpcError as exc:
---> [78](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:78) raise exceptions.from_grpc_error(exc) from exc

InvalidArgument: 400 Invalid Metrics format.
```

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。