[Feature] Implementation of a TrainingHealthMonitor for real-time gradient norm tracking.

🚀 Feature

Description

While the current suite of Ignite handlers covers a wide range of training needs, monitoring loss alone can be insufficient for detecting early-stage instability in complex models, such as transformers or tasks involving curriculum-based learning.

I would like to propose the addition of a TrainingHealthMonitor utility. In my recent work with logic-based transformer prototypes, I found that tracking the L2 gradient norm at the iteration level was essential for identifying "entropy collapse" and optimization stress during curriculum stage transitions before the model diverged.

Proposed Implementation

I suggest a handler that hooks into the ITERATION_COMPLETED event:

• Threshold Alerts: Automatically log a warning or trigger a user-defined action if the L2 gradient norm exceeds a specific threshold.

• Optimization Visibility: Provide researchers with a real-time heartbeat of the training process, which is particularly valuable when tuning learning rates or stabilizing curriculum jumps.

• Ignite Native Integration: Designed to attach seamlessly to the Engine and work alongside existing handlers like the ProgressBar.

Technical Justification

While safety measures like gradient clipping (as discussed in #375) prevent catastrophic failures, they often mask underlying instability. A dedicated health monitor provides the transparency needed to debug optimization issues proactively.

I am very interested in contributing this feature to the library and would appreciate the maintainers' feedback on this proposal.

Start by reviewing Ignite's existing handlers, the Engine integration, and the ProgressBar handler to understand how ITERATION_COMPLETED is used. Define the monitor's gradient-norm threshold behavior and user-defined action interface, then verify that it reports optimization instability in real time without interfering with training.