[Monitor] Alert System Refine

Working items about Alerts:

Email clarity

• P2 Alerts severity should be customized by clusters. e.x., NodeNotReady is not a serious issue in wu2 cluster. Admin should have the ability to modify configmap easily.
• P2 auto-set repeat_interval / group_interval by alert severity

Webportal:

• P2 filter alert by user/admin, show only related alerts to users
• P2 colorize the alerts by severity level, refer to https://hu.pinterest.com/pin/556194622715526398/ (default alert severity)

Delete script

• P2 lack delete storage scripts

Misleading alerts:

• P2: NodeGPUCountChanged: configured GPU count is read from layout.yml, however the configured GPU count is not right. Included in #5100
• P3 Misleading Worker Node Alerts:
  • NodeMemoryUsage #2760;
  • NodeCPUUsage #2762
  • NodeFilesystemUsage, NodeDiskPressure, NodeOutOfDisk,NodeNotReady, AzureAgentConsumeTooMuch
  • Alert only on master node; Questions: what about worker node ?
• P3: Issue @scarlett2018 reported in email: For a low priority VMSS bed, nodes get preempted by Azure and get back when available every day. “Node not ready” alerts are too frequent to be triggered. (and seems no actions ops can take for these type of alerts). #3806 We may need a add an exporter for this

Alerts Summary

• P3 alert logging & daily summary / email to admin; Consider : https://github.com/microsoft/pai/issues/4944
  Questions: Where to save the statistics ?

Alert coverage

• P3 Add more alerts for PAI service, such as CPU usage, Memory usage, Disk usage. Make sure PAI service not consume too much resource (especially daemon service)

Other Candidates

• P5 alert for multi-cluster
• P3 auto resolve alerts. Such as run privileged diagnostic pod in worker node. And run command to solve the current alerts

DONE:

Duplicated alerts: #5052

• P1 Inhibition : some alerts are included in more high level alerts:
  nodeNotReady-> PaiServicePodNotRunning|PaiServicePodNotReady (in the nodes)
• P1 PaiServicePodNotRunning|PaiServicePodNotReady and PaiServiceNotUp Overlap

Admin Experiment

• P1 Alert severity level, group by level; critical/error/warn/info. #5055
• P1 Email template refine: #5064
  • different templates for users;
  • Instruction on how to deal with the alert: add link to openpai handbook->troubleshooting;
  • Kill job alert: https://github.com/microsoft/pai/issues/5021
  • Allow users adding customized email template without rebuild alertmanager

Alerts Definition Not Accurate

• P1 NodeGpuCountChanged: change this metric to compare current GPU count with configured GPU count. #5072