`ignite.metrics.Metric` for all use-cases: Expand Metric's arguments by attachment events

Issue

ignite.metrics.Metric currently only supports the calculation (and therefore also the visualization/tensorboard logging) of metrics at each completed epoch. In regular use-cases, e.g. to determine/visualize the optimum iteration for early stopping, it is required to return the loss metric after each iteration or at a customized event. This is currently always related to overriding Metric.attach(). So currently I experienced the number of use-cases to be very limited without overriding the source code all the time.

Solution

Add all events used to attach the methods of Metric to the arguments with default values referring to the current attachment events. The default values of the added event arguments will guarantee the backward compatibility of existing code and keep boilerplate code at the same level for "epoch-metrics". For any other desired/required use-case, the event arguments will provide updating, calculating and outputting the metric to engine.metrics at any desired/required event.
This would of course require modifying all inheriting classes of Metric (e.g. Loss, Accuracy, LambdaMetric and their base classes) but the modifications are simple and straight forward.

Code suggestion

Here a possible implementation:

from abc import ABCMeta, abstractmethod
from ignite._six import with_metaclass
from ignite.engine import Events
import torch


class Metric(with_metaclass(ABCMeta, object)):
    """
    Base class for all Metrics.

    Args:
        output_transform (callable, optional): a callable that is used to transform the
            :class:`~ignite.engine.Engine`'s `process_function`'s output into the
            form expected by the metric. This can be useful if, for example, you have a multi-output model and
            you want to compute the metric with respect to one of the outputs.
        started_event (<enum Events>): event from which on the metric should be calculated
        iteration_completed_event (<enum Events>): event at which intermediate metric calculations (`self.update()`)
            are executed from current model outputs, e.g. for a mean loss metric this would refer to adding
            the current model loss output after each iteration to a summed loss and increasing to count of losses added.
        completed_event (<enum Events>): event at which the metric value us calculated and written to
            `engine.state.metrics[metric_name]`. E.g. for mean loss metric calculation the summed output losses
            of each iterations is devided by the number of iterations and outputed to `trainer.state.metrics['loss'].
    """

    def __init__(self, output_transform=lambda x: x, started_event=Events.EPOCH_STARTED,
                 iteration_completed_event=Events.ITERATION_COMPLETED, completed_event=Events.EPOCH_COMPLETED):
        self._output_transform = output_transform
        self.started_event = started_event
        self.iteration_completed_event = iteration_completed_event
        self.completed_event = completed_event
        self.reset()

    @abstractmethod
    def reset(self):
        """
        Resets the metric to it's initial state.

        This is called at the start of each epoch.
        """
        pass

    @abstractmethod
    def update(self, output):
        """
        Updates the metric's state using the passed batch output.

        This is called once for each batch.

        Args:
            output: the is the output from the engine's process function.
        """
        pass

    @abstractmethod
    def compute(self):
        """
        Computes the metric based on it's accumulated state.

        This is called at the end of each epoch.

        Returns:
            Any: the actual quantity of interest.

        Raises:
            NotComputableError: raised when the metric cannot be computed.
        """
        pass

    def started(self, engine):
        self.reset()

    @torch.no_grad()
    def iteration_completed(self, engine):
        output = self._output_transform(engine.state.output)
        self.update(output)

    def completed(self, engine, name):
        result = self.compute()
        if torch.is_tensor(result) and len(result.shape) == 0:
            result = result.item()
        engine.state.metrics[name] = result

    def attach(self, engine, name):
        if not engine.has_event_handler(self.started, self.started_event):
            engine.add_event_handler(self.started_event, self.started)
        if not engine.has_event_handler(self.iteration_completed, self.iteration_completed_event):
            engine.add_event_handler(self.iteration_completed_event, self.iteration_completed)
        # `self.completed()` is always executed at the end, so it should be added to `engine` at the end
        engine.add_event_handler(self.completed_event, self.completed, name)

    def __add__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x + y, self, other)

    def __radd__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x + y, other, self)

    def __sub__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x - y, self, other)

    def __rsub__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x - y, other, self)

    def __mul__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x * y, self, other)

    def __rmul__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x * y, other, self)

    def __pow__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x ** y, self, other)

    def __rpow__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x ** y, other, self)

    def __mod__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x % y, self, other)

    def __div__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x.__div__(y), self, other)

    def __rdiv__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x.__div__(y), other, self)

    def __truediv__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x.__truediv__(y), self, other)

    def __rtruediv__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x.__truediv__(y), other, self)

    def __floordiv__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x // y, self, other)

    def __getattr__(self, attr):
        from ignite.metrics import MetricsLambda

        def fn(x, *args, **kwargs):
            return getattr(x, attr)(*args, **kwargs)

        def wrapper(*args, **kwargs):
            return MetricsLambda(fn, self, *args, **kwargs)
        return wrapper

    def __getitem__(self, index):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x: x[index], self)

Start with ignite/metrics/metric.py and inspect the named inheriting classes, including Loss, Accuracy, LambdaMetric, and their base classes. Add configurable attachment events while preserving the current epoch-event defaults, and verify that custom iteration or other event attachments update and publish metrics as described.