Implement Test and Validation Set Loss

I propose creating a tracker to capture a stable loss measurement as proposed here: https://github.com/spacepxl/demystifying-sd-finetuning

Effectively, at regular intervals during training, a preselected image (or batch of images I imagine) and a preselected noise seed are used to calculate the loss. This ensures that over the course of training, the loss recorded can accurately show the progress of the training run.

By also incorporating a holdout set to use for calculating validation loss, this allows a proper evaluation of what point the model begins to overtrain.

Start by reading the linked proposal and locating the training loss-tracking entry points; the issue names no files or tests. Define how fixed images, noise seeds, sampling intervals, and a holdout set should be represented, then verify that training records both stable and validation loss over time.

Written by the indexing model from the issue text.