AI-Hypercomputer / AI-Hypercomputer/google-cloud-mldiagnostics
Model-metrics dashboard charts require Log Analytics on _Default bucket (undocumented; misleading "0 time series" error)
- Linguagem predominante
- Python
- Estrelas
- 14
- Forks
- 4
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
### Summary
The Model-metrics (and performance-metrics) **time series charts** in the ML Diagnostics dashboard (Cluster Director → Diagnostics, and GKE → AI/ML → Diagnostics) render their data by running a **Log Analytics SQL query** against the project's `_Default` log bucket. If **Log Analytics is not enabled** on that bucket, every model/performance-metric widget fails — even though the metric data is present and correct in Cloud Logging, the MLRun is healthy, and Profiles + System metrics render fine.
This prerequisite is **not documented**, and the failure mode is a **generic, misleading error**.
### Impact
- **Symptom:** Model-metrics widgets show *"Something went wrong in the backend"* / *"0 time series"*, while Profiles and System metrics render normally.
- The metric log entries (`log_name=".../ml_diagnostics_metric"`) are present and correctly shaped in Cloud Logging the whole time — the data is fine; only the chart's query path fails.
- Directly querying reveals the real cause: the widget runs Log Analytics SQL over `` `PROJECT.global._Default._Default` `` and the bucket returns *"Bucket .../buckets/_Default ... does not have Log Analytics enabled."*
### Root cause
`_Default` had `analyticsEnabled: false`. The dashboard's model-metric charts depend on Log Analytics being enabled on that bucket, but nothing in the managed onboarding (`gcloud container clusters update --enable-managed-mldiagnostics`) enables it, and the docs don't mention it.
### Fix (for users)
Enable Log Analytics on the `_Default` bucket (non-destructive upgrade — existing logs stay queryable, no added cost, cannot be disabled):
```
gcloud logging buckets update _Default --location=global --enable-analytics --project=PROJECT
```
Requires `logging.buckets.update` (e.g. `roles/logging.configWriter`).
### ⚠️ Important gotcha: Log Analytics does not backfill
Enabling Log Analytics only indexes logs **ingested after** the flag is turned on. **Runs that completed before enablement will still show empty charts** — their rows aren't in the analytics view. After enabling, launch a **new** run to see charts populate. This is easy to misdiagnose as "the fix didn't work."
### Requests
1. **Auto-enable (or gate on) Log Analytics** as part of managed ML Diagnostics onboarding, since the model-metric dashboard hard-depends on it.
2. **Improve the failure UX** — a Log-Analytics-disabled bucket should surface an actionable error (e.g. *"Log Analytics is not enabled on the _Default bucket; enable it to view model metrics"*) instead of the generic *"Something went wrong in the backend / 0 time series."*
3. **Document the prerequisite** in the README (Viewing Logs, Metrics and Profiles) and setup guide — the exact `gcloud logging buckets update ... --enable-analytics` command, the least-privilege role, and the no-backfill behavior.
I'll attach a small README docs PR covering (3).
### Environment
`google-cloud-mldiagnostics` 1.0.3, JAX 0.10.0, GKE workload on TPU (plain-SPMD), managed-mldiagnostics enabled (connection-operator + injection-webhook present). Confirmed: enabling Log Analytics on `_Default`, then running a fresh MLRun, makes the model-metric charts render natively (loss / learning rate / throughput / tflops, over Time and over Steps).
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.