googleapis / googleapis/python-aiplatform

ModelServiceClient.import_model_evaluation does not work as expected

未关闭
#5,570 1 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
api: vertex-ai
主要语言
Python
星标
905
派生
465
平均合并
1 天 13 小时
30 天内合并 PR
44

描述

I wanted to upload a model evaluation following the gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml and the steps described in https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb. The code snippet in the tutorial works fine, but I want to add a "confusionMatrix", as defined by the metric schema. The same code snippet now doesn't work anymore, and the import_model_evaluation returns a "400 Invalid Metrics Format" error.

To find out the issue, I have tried to print out a model evaluation created by a automl tabular job and upload said evaluation to a custom model, but this also results in the same 400 error. The automl evaluation is reported to follow the metric schema, as it is returned as its metricsSchemaUri. It seems to me the reported evaluation results do not follow the metric schema, as the "rows" contain arrays of arrays, instead of the expected arrays of objects of arrays required by the schema yaml. I have attached a log file with the full exported evaluation metric used for the test [automl.log](https://github.com/user-attachments/files/21312310/automl.log)

I have also tested uploading a reduced form of the evaluation metrics dictionary in two formats:

```
{
"confusionMatrix": {
"annotationSpecs": [
{"displayName": "Husband"},
{"displayName": "Not-in-family"},
{"displayName": "Own-child"},
{"displayName": "Wife"},
{"displayName": "Unmarried"},
{"displayName": "Other-relative"},
],
"rows": [
{"row": [1614, 0, 0, 0, 0, 0]},
{"row": [2, 805, 1, 124, 96, 1]},
{"row": [11, 52, 2, 37, 12, 4]},
{"row": [5, 188, 2, 359, 30, 6]},
{"row": [0, 202, 1, 28, 187, 0]},
{"row": [0, 0, 0, 0, 0, 186]},
],
},
"auPrc": 0.8,
"auRoc": 0.8,
"logLoss": 0.5,
}
```

and

```
{
"confusionMatrix": {
"annotationSpecs": [
{"displayName": "Husband"},
{"displayName": "Not-in-family"},
{"displayName": "Own-child"},
{"displayName": "Wife"},
{"displayName": "Unmarried"},
{"displayName": "Other-relative"},
],
"rows": [
[1614, 0, 0, 0, 0, 0],
[2, 805, 1, 124, 96, 1],
[11, 52, 2, 37, 12, 4],
[5, 188, 2, 359, 30, 6],
[0, 202, 1, 28, 187, 0],
[0, 0, 0, 0, 0, 186],
],
},
"auPrc": 0.8,
"auRoc": 0.8,
"logLoss": 0.5,
}
```

Both do not work and will return a 400.

#### Environment details

- OS type and version: MacOS Sequoia 15.5
- Python version: 3.12.11
- pip version: 25.1.1
- `google-cloud-aiplatform` version:
Name: google-cloud-aiplatform
Version: 1.92.0
Summary: Vertex AI API client library
Home-page: https://github.com/googleapis/python-aiplatform
Author: Google LLC
Author-email: googleapis-packages@google.com
License: Apache 2.0
Location: -
Requires: docstring-parser, google-api-core, google-auth, google-cloud-bigquery, google-cloud-resource-manager, google-cloud-storage, google-genai, packaging, proto-plus, protobuf, pydantic, shapely, typing-extensions
Required-by:

#### Steps to reproduce

1. Follow https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb#scrollTo=852be5e5f964
2. Instead of just uploading the small evaluation {"logLoss": ...,"auRoc": ...} use the provided metric dicts with confusionMatrix
4. Observe the metric invalid error

#### Stack trace
```
---------------------------------------------------------------------------
_InactiveRpcError Traceback (most recent call last)
File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:76, in _wrap_unary_errors..error_remapped_callable(*args, **kwargs)
75 try:
---> [76](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:76) return callable_(*args, **kwargs)
77 except grpc.RpcError as exc:

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:277, in _UnaryUnaryMultiCallable.__call__(self, request, timeout, metadata, credentials, wait_for_ready, compression)
268 def __call__(
269 self,
270 request: Any,
(...) 275 compression: Optional[grpc.Compression] = None,
276 ) -> Any:
--> [277](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:277) response, ignored_call = self._with_call(
278 request,
279 timeout=timeout,
280 metadata=metadata,
281 credentials=credentials,
282 wait_for_ready=wait_for_ready,
283 compression=compression,
284 )
285 return response

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:332, in _UnaryUnaryMultiCallable._with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
329 call = self._interceptor.intercept_unary_unary(
330 continuation, client_call_details, request
331 )
--> [332](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:332) return call.result(), call

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:440, in _InactiveRpcError.result(self, timeout)
439 """See grpc.Future.result."""
--> [440](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:440) raise self

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:315, in _UnaryUnaryMultiCallable._with_call..continuation(new_details, request)
314 try:
--> [315](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_interceptor.py:315) response, call = self._thunk(new_method).with_call(
316 request,
317 timeout=new_timeout,
318 metadata=new_metadata,
319 credentials=new_credentials,
320 wait_for_ready=new_wait_for_ready,
321 compression=new_compression,
322 )
323 return _UnaryOutcome(response, call)

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1198, in _UnaryUnaryMultiCallable.with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
1192 (
1193 state,
1194 call,
1195 ) = self._blocking(
1196 request, timeout, metadata, credentials, wait_for_ready, compression
1197 )
-> [1198](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1198) return _end_unary_response_blocking(state, call, True, None)

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1006, in _end_unary_response_blocking(state, call, with_call, deadline)
1005 else:
-> [1006](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/grpc/_channel.py:1006) raise _InactiveRpcError(state)

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Invalid Metrics format."
debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B2a00:1450:4001:80f::200a%5D:443 {created_time:"2025-07-18T09:18:03.606562+02:00", grpc_status:3, grpc_message:"Invalid Metrics format."}"
>

The above exception was the direct cause of the following exception:

InvalidArgument Traceback (most recent call last)
Cell In[64], [line 14](vscode-notebook-cell:?execution_count=64&line=14)
6 model_eval = ModelEvaluation(
7 display_name=f"{EXPERIMENT_NAME}-eval",
8 metrics_schema_uri="gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
9 metrics=fixed_dict,
10 )
11 client = ModelServiceClient(
12 client_options={"api_endpoint": f"{LOCATION}-aiplatform.googleapis.com"}
13 )
---> [14](vscode-notebook-cell:?execution_count=64&line=14) client.import_model_evaluation(parent=model.resource_name, model_evaluation=model_eval)

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/model_service/client.py:2592, in ModelServiceClient.import_model_evaluation(self, request, parent, model_evaluation, retry, timeout, metadata)
2589 self._validate_universe_domain()
2591 # Send the request.
-> [2592](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/model_service/client.py:2592) response = rpc(
2593 request,
2594 retry=retry,
2595 timeout=timeout,
2596 metadata=metadata,
2597 )
2599 # Done; return the response.
2600 return response

File ~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py:131, in _GapicCallable.__call__(self, timeout, retry, compression, *args, **kwargs)
128 if self._compression is not None:
129 kwargs["compression"] = compression
--> [131](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py:131) return wrapped_func(*args, **kwargs)

File ~/Projekte/Sandbox/sandbox/autogluon/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:78, in _wrap_unary_errors..error_remapped_callable(*args, **kwargs)
76 return callable_(*args, **kwargs)
77 except grpc.RpcError as exc:
---> [78](~/Projekte/Sandbox/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:78) raise exceptions.from_grpc_error(exc) from exc

InvalidArgument: 400 Invalid Metrics format.
```

贡献指南

打开贡献指南

调研方向

从 ModelServiceClient.import_model_evaluation 开始,使用提供的 confusionMatrix 字典和分类指标架构 URI 重现 400 错误。将教程 payload 与导出的 AutoML 评估结果进行比较,并检查相关测试或验证路径。完成标准是能够导入有效的 confusionMatrix,并且回归用例通过。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
api, machine-learning
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。