AI-Hypercomputer / AI-Hypercomputer/google-cloud-mldiagnostics

Model-metrics dashboard charts require Log Analytics on _Default bucket (undocumented; misleading "0 time series" error)

Abierto
#6 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
14
Forks
4
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

### Summary

The Model-metrics (and performance-metrics) **time series charts** in the ML Diagnostics dashboard (Cluster Director → Diagnostics, and GKE → AI/ML → Diagnostics) render their data by running a **Log Analytics SQL query** against the project's `_Default` log bucket. If **Log Analytics is not enabled** on that bucket, every model/performance-metric widget fails — even though the metric data is present and correct in Cloud Logging, the MLRun is healthy, and Profiles + System metrics render fine.

This prerequisite is **not documented**, and the failure mode is a **generic, misleading error**.

### Impact

- **Symptom:** Model-metrics widgets show *"Something went wrong in the backend"* / *"0 time series"*, while Profiles and System metrics render normally.
- The metric log entries (`log_name=".../ml_diagnostics_metric"`) are present and correctly shaped in Cloud Logging the whole time — the data is fine; only the chart's query path fails.
- Directly querying reveals the real cause: the widget runs Log Analytics SQL over `` `PROJECT.global._Default._Default` `` and the bucket returns *"Bucket .../buckets/_Default ... does not have Log Analytics enabled."*

### Root cause

`_Default` had `analyticsEnabled: false`. The dashboard's model-metric charts depend on Log Analytics being enabled on that bucket, but nothing in the managed onboarding (`gcloud container clusters update --enable-managed-mldiagnostics`) enables it, and the docs don't mention it.

### Fix (for users)

Enable Log Analytics on the `_Default` bucket (non-destructive upgrade — existing logs stay queryable, no added cost, cannot be disabled):

```
gcloud logging buckets update _Default --location=global --enable-analytics --project=PROJECT
```

Requires `logging.buckets.update` (e.g. `roles/logging.configWriter`).

### ⚠️ Important gotcha: Log Analytics does not backfill

Enabling Log Analytics only indexes logs **ingested after** the flag is turned on. **Runs that completed before enablement will still show empty charts** — their rows aren't in the analytics view. After enabling, launch a **new** run to see charts populate. This is easy to misdiagnose as "the fix didn't work."

### Requests

1. **Auto-enable (or gate on) Log Analytics** as part of managed ML Diagnostics onboarding, since the model-metric dashboard hard-depends on it.
2. **Improve the failure UX** — a Log-Analytics-disabled bucket should surface an actionable error (e.g. *"Log Analytics is not enabled on the _Default bucket; enable it to view model metrics"*) instead of the generic *"Something went wrong in the backend / 0 time series."*
3. **Document the prerequisite** in the README (Viewing Logs, Metrics and Profiles) and setup guide — the exact `gcloud logging buckets update ... --enable-analytics` command, the least-privilege role, and the no-backfill behavior.

I'll attach a small README docs PR covering (3).

### Environment

`google-cloud-mldiagnostics` 1.0.3, JAX 0.10.0, GKE workload on TPU (plain-SPMD), managed-mldiagnostics enabled (connection-operator + injection-webhook present). Confirmed: enabling Log Analytics on `_Default`, then running a fresh MLRun, makes the model-metric charts render natively (loss / learning rate / throughput / tflops, over Time and over Steps).

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.