AI-Hypercomputer / AI-Hypercomputer/google-cloud-mldiagnostics

machinelearning_run() 400 INVALID_ARGUMENT: GKE createTime passed verbatim to proto Timestamp (not RFC 3339)

未关闭
#2 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
14
派生
4
PR 合并指标
30 天内没有已合并 PR

描述

## Summary
`machinelearning_run()` fails with **HTTP 400 INVALID_ARGUMENT** at the control-plane create POST when the GKE workload `createTime` is not strict RFC 3339. The SDK passes `workload_details["creation-timestamp"]` verbatim into the `google.protobuf.Timestamp` field `workload_details.gke.create_time`.

## Environment
- `google-cloud-mldiagnostics` 1.0.2 / 1.0.3
- GKE TPU (tpu7x, GKE 1.35.3-gke.1522000), single-controller Pathways (JAX 0.10.0)
- `--enable-managed-mldiagnostics` on; injection-webhook injecting `GKE_DIAGON_*`

## Error (verbatim)
```
POST .../machineLearningRuns?machine_learning_run_id=
400 INVALID_ARGUMENT
fieldViolations[0]:
field: machine_learning_run.workload_details.gke.create_time
description: "Invalid value at '...gke.create_time' (google.protobuf.Timestamp),
Field 'createTime', Illegal timestamp format; timestamps must end with 'Z'
or have a valid timezone offset."
```

## Root cause
`clients/control_plane_client.py` (around L259-261):
```python
creation_timestamp = workload_details.get("creation-timestamp")
if creation_timestamp:
gke_workload_details["createTime"] = creation_timestamp # verbatim → proto Timestamp
```
No RFC-3339 normalization. The source `creation-timestamp` (from `GKE_DIAGON_METADATA`) isn't guaranteed to be RFC 3339. Also `host_utils.get_identifier` itself formats timestamps as `"%Y%m%d-%H%M%S"` (no `Z`), so this is latent even for intended inputs.

## Fix
PR #1 adds a `_normalize_rfc3339()` helper applied at the `createTime` assignment (best-effort; unparseable values pass through). Verified: the 400 cleared and the MLRun was created.

— Reported via Navi on behalf of @lokic233 (Meta MRS-CE).

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。