AI-Hypercomputer / AI-Hypercomputer/google-cloud-mldiagnostics

Model-metrics dashboard charts require Log Analytics on _Default bucket (undocumented; misleading "0 time series" error)

Aperta
#6 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
14
Fork
4
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

### Summary

The Model-metrics (and performance-metrics) **time series charts** in the ML Diagnostics dashboard (Cluster Director → Diagnostics, and GKE → AI/ML → Diagnostics) render their data by running a **Log Analytics SQL query** against the project's `_Default` log bucket. If **Log Analytics is not enabled** on that bucket, every model/performance-metric widget fails — even though the metric data is present and correct in Cloud Logging, the MLRun is healthy, and Profiles + System metrics render fine.

This prerequisite is **not documented**, and the failure mode is a **generic, misleading error**.

### Impact

- **Symptom:** Model-metrics widgets show *"Something went wrong in the backend"* / *"0 time series"*, while Profiles and System metrics render normally.
- The metric log entries (`log_name=".../ml_diagnostics_metric"`) are present and correctly shaped in Cloud Logging the whole time — the data is fine; only the chart's query path fails.
- Directly querying reveals the real cause: the widget runs Log Analytics SQL over `` `PROJECT.global._Default._Default` `` and the bucket returns *"Bucket .../buckets/_Default ... does not have Log Analytics enabled."*

### Root cause

`_Default` had `analyticsEnabled: false`. The dashboard's model-metric charts depend on Log Analytics being enabled on that bucket, but nothing in the managed onboarding (`gcloud container clusters update --enable-managed-mldiagnostics`) enables it, and the docs don't mention it.

### Fix (for users)

Enable Log Analytics on the `_Default` bucket (non-destructive upgrade — existing logs stay queryable, no added cost, cannot be disabled):

```
gcloud logging buckets update _Default --location=global --enable-analytics --project=PROJECT
```

Requires `logging.buckets.update` (e.g. `roles/logging.configWriter`).

### ⚠️ Important gotcha: Log Analytics does not backfill

Enabling Log Analytics only indexes logs **ingested after** the flag is turned on. **Runs that completed before enablement will still show empty charts** — their rows aren't in the analytics view. After enabling, launch a **new** run to see charts populate. This is easy to misdiagnose as "the fix didn't work."

### Requests

1. **Auto-enable (or gate on) Log Analytics** as part of managed ML Diagnostics onboarding, since the model-metric dashboard hard-depends on it.
2. **Improve the failure UX** — a Log-Analytics-disabled bucket should surface an actionable error (e.g. *"Log Analytics is not enabled on the _Default bucket; enable it to view model metrics"*) instead of the generic *"Something went wrong in the backend / 0 time series."*
3. **Document the prerequisite** in the README (Viewing Logs, Metrics and Profiles) and setup guide — the exact `gcloud logging buckets update ... --enable-analytics` command, the least-privilege role, and the no-backfill behavior.

I'll attach a small README docs PR covering (3).

### Environment

`google-cloud-mldiagnostics` 1.0.3, JAX 0.10.0, GKE workload on TPU (plain-SPMD), managed-mldiagnostics enabled (connection-operator + injection-webhook present). Confirmed: enabling Log Analytics on `_Default`, then running a fresh MLRun, makes the model-metric charts render natively (loss / learning rate / throughput / tflops, over Time and over Steps).

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.