googleapis / googleapis/python-aiplatform

ModelServiceClient.import_model_evaluation does not work as expected

Aperta
#5,570 1 commento 1 reazione 0 assegnatari Vedi su GitHub
api: vertex-ai
Lingua principale
Python
Stelle
905
Fork
465
Merge medio
1g 13h
PR unite (30g)
44

Descrizione

I wanted to upload a model evaluation following the gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml and the steps described in https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb. The code snippet in the tutorial works fine, but I want to add a "confusionMatrix", as defined by the metric schema. The same code snippet now doesn't work anymore, and the import_model_evaluation returns a "400 Invalid Metrics Format" error.

To find out the issue, I have tried to print out a model evaluation created by a automl tabular job and upload said evaluation to a custom model, but this also results in the same 400 error. The automl evaluation is reported to follow the metric schema, as it is returned as its metricsSchemaUri. It seems to me the reported evaluation results do not follow the metric schema, as the "rows" contain arrays of arrays, instead of the expected arrays of objects of arrays required by the schema yaml. I have attached a log file with the full exported evaluation metric used for the test [automl.log](https://github.com/user-attachments/files/21312310/automl.log)

I have also tested uploading a reduced form of the evaluation metrics dictionary in two formats:

```
{
"confusionMatrix": {
"annotationSpecs": [
{"displayName": "Husband"},
{"displayName": "Not-in-family"},
{"displayName": "Own-child"},
{"displayName": "Wife"},
{"displayName": "Unmarried"},
{"displayName": "Other-relative"},
],
"rows": [
{"row": [1614, 0, 0, 0, 0, 0]},
{"row": [2, 805, 1, 124, 96, 1]},
{"row": [11, 52, 2, 37, 12, 4]},
{"row": [5, 188, 2, 359, 30, 6]},
{"row": [0, 202, 1, 28, 187, 0]},
{"row": [0, 0, 0, 0, 0, 186]},
],
},
"auPrc": 0.8,
"auRoc": 0.8,
"logLoss": 0.5,
}
```

and

```
{
"confusionMatrix": {
"annotationSpecs": [
{"displayName": "Husband"},
{"displayName": "Not-in-family"},
{"displayName": "Own-child"},
{"displayName": "Wife"},
{"displayName": "Unmarried"},
{"displayName": "Other-relative"},
],
"rows": [
[1614, 0, 0, 0, 0, 0],
[2, 805, 1, 124, 96, 1],
[11, 52, 2, 37, 12, 4],
[5, 188, 2, 359, 30, 6],
[0, 202, 1, 28, 187, 0],
[0, 0, 0, 0, 0, 186],
],
},
"auPrc": 0.8,
"auRoc": 0.8,
"logLoss": 0.5,
}
```

Both do not work and will return a 400.

#### Environment details

- OS type and version: MacOS Sequoia 15.5
- Python version: 3.12.11
- pip version: 25.1.1
- `google-cloud-aiplatform` version:
Name: google-cloud-aiplatform
Version: 1.92.0
Summary: Vertex AI API client library
Home-page: https://github.com/googleapis/python-aiplatform
Author: Google LLC
Author-email: googleapis-packages@google.com
License: Apache 2.0
Location: -
Requires: docstring-parser, google-api-core, google-auth, google-cloud-bigquery, google-cloud-resource-manager, google-cloud-storage, google-genai, packaging, proto-plus, protobuf, pydantic, shapely, typing-extensions
Required-by:

#### Steps to reproduce

1. Follow https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb#scrollTo=852be5e5f964
2. Instead of just uploading the small evaluation {"logLoss": ...,"auRoc": ...} use the provided metric dicts with confusionMatrix
4. Observe the metric invalid error

#### Stack trace
```
---------------------------------------------------------------------------
_InactiveRpcError Traceback (most recent call last)
File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:76, in _wrap_unary_errors..error_remapped_callable(*args, **kwargs)
75 try:
---> [76](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:76) return callable_(*args, **kwargs)
77 except grpc.RpcError as exc:

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:277, in _UnaryUnaryMultiCallable.__call__(self, request, timeout, metadata, credentials, wait_for_ready, compression)
268 def __call__(
269 self,
270 request: Any,
(...) 275 compression: Optional[grpc.Compression] = None,
276 ) -> Any:
--> [277](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:277) response, ignored_call = self._with_call(
278 request,
279 timeout=timeout,
280 metadata=metadata,
281 credentials=credentials,
282 wait_for_ready=wait_for_ready,
283 compression=compression,
284 )
285 return response

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:332, in _UnaryUnaryMultiCallable._with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
329 call = self._interceptor.intercept_unary_unary(
330 continuation, client_call_details, request
331 )
--> [332](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:332) return call.result(), call

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:440, in _InactiveRpcError.result(self, timeout)
439 """See grpc.Future.result."""
--> [440](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:440) raise self

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:315, in _UnaryUnaryMultiCallable._with_call..continuation(new_details, request)
314 try:
--> [315](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:315) response, call = self._thunk(new_method).with_call(
316 request,
317 timeout=new_timeout,
318 metadata=new_metadata,
319 credentials=new_credentials,
320 wait_for_ready=new_wait_for_ready,
321 compression=new_compression,
322 )
323 return _UnaryOutcome(response, call)

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1198, in _UnaryUnaryMultiCallable.with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
1192 (
1193 state,
1194 call,
1195 ) = self._blocking(
1196 request, timeout, metadata, credentials, wait_for_ready, compression
1197 )
-> [1198](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1198) return _end_unary_response_blocking(state, call, True, None)

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1006, in _end_unary_response_blocking(state, call, with_call, deadline)
1005 else:
-> [1006](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1006) raise _InactiveRpcError(state)

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Invalid Metrics format."
debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B2a00:1450:4001:80f::200a%5D:443 {created_time:"2025-07-18T09:18:03.606562+02:00", grpc_status:3, grpc_message:"Invalid Metrics format."}"
>

The above exception was the direct cause of the following exception:

InvalidArgument Traceback (most recent call last)
Cell In[64], [line 14](vscode-notebook-cell:?execution_count=64&line=14)
6 model_eval = ModelEvaluation(
7 display_name=f"{EXPERIMENT_NAME}-eval",
8 metrics_schema_uri="gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
9 metrics=fixed_dict,
10 )
11 client = ModelServiceClient(
12 client_options={"api_endpoint": f"{LOCATION}-aiplatform.googleapis.com"}
13 )
---> [14](vscode-notebook-cell:?execution_count=64&line=14) client.import_model_evaluation(parent=model.resource_name, model_evaluation=model_eval)

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/model_service/client.py:2592, in ModelServiceClient.import_model_evaluation(self, request, parent, model_evaluation, retry, timeout, metadata)
2589 self._validate_universe_domain()
2591 # Send the request.
-> [2592](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/model_service/client.py:2592) response = rpc(
2593 request,
2594 retry=retry,
2595 timeout=timeout,
2596 metadata=metadata,
2597 )
2599 # Done; return the response.
2600 return response

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py:131, in _GapicCallable.__call__(self, timeout, retry, compression, *args, **kwargs)
128 if self._compression is not None:
129 kwargs["compression"] = compression
--> [131](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py:131) return wrapped_func(*args, **kwargs)

File ~/Projekte/Sandbox/sandbox/autogluon/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:78, in _wrap_unary_errors..error_remapped_callable(*args, **kwargs)
76 return callable_(*args, **kwargs)
77 except grpc.RpcError as exc:
---> [78](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:78) raise exceptions.from_grpc_error(exc) from exc

InvalidArgument: 400 Invalid Metrics format.
```

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.