AI-Hypercomputer / AI-Hypercomputer/google-cloud-mldiagnostics
machinelearning_run() 400 INVALID_ARGUMENT: GKE createTime passed verbatim to proto Timestamp (not RFC 3339)
- Linguagem predominante
- Python
- Estrelas
- 14
- Forks
- 4
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
## Summary
`machinelearning_run()` fails with **HTTP 400 INVALID_ARGUMENT** at the control-plane create POST when the GKE workload `createTime` is not strict RFC 3339. The SDK passes `workload_details["creation-timestamp"]` verbatim into the `google.protobuf.Timestamp` field `workload_details.gke.create_time`.
## Environment
- `google-cloud-mldiagnostics` 1.0.2 / 1.0.3
- GKE TPU (tpu7x, GKE 1.35.3-gke.1522000), single-controller Pathways (JAX 0.10.0)
- `--enable-managed-mldiagnostics` on; injection-webhook injecting `GKE_DIAGON_*`
## Error (verbatim)
```
POST .../machineLearningRuns?machine_learning_run_id=
400 INVALID_ARGUMENT
fieldViolations[0]:
field: machine_learning_run.workload_details.gke.create_time
description: "Invalid value at '...gke.create_time' (google.protobuf.Timestamp),
Field 'createTime', Illegal timestamp format; timestamps must end with 'Z'
or have a valid timezone offset."
```
## Root cause
`clients/control_plane_client.py` (around L259-261):
```python
creation_timestamp = workload_details.get("creation-timestamp")
if creation_timestamp:
gke_workload_details["createTime"] = creation_timestamp # verbatim → proto Timestamp
```
No RFC-3339 normalization. The source `creation-timestamp` (from `GKE_DIAGON_METADATA`) isn't guaranteed to be RFC 3339. Also `host_utils.get_identifier` itself formats timestamps as `"%Y%m%d-%H%M%S"` (no `Z`), so this is latent even for intended inputs.
## Fix
PR #1 adds a `_normalize_rfc3339()` helper applied at the `createTime` assignment (best-effort; unparseable values pass through). Verified: the 400 cleared and the MLRun was created.
— Reported via Navi on behalf of @lokic233 (Meta MRS-CE).
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.