Representative bleu scores.

We often train a model from a Back Translation to the translation and use that to translate from a published Bible. Recently I've started training models with a mixed source: part published Bible and part Back Translation. Those give a bleu score for the translation from the source that we will use to create drafts for the translation team.

It would be ideal if we could specify a list of source texts which are likely candidates and have the system draft the set of test verses from each of those and calculate the bleu scores. That would give us a range of bleu scores that are representative of the performance of the model as it will be used.

This would also give us a way to compare models with the same source text even if that wasn't part of the initial test set.

A standalone option to run these on existing models would be helpful. Clowder would be useful as a way of running them in an integrated way with a training run.

Start by locating the existing BLEU evaluation flow for test verses and the entry point used to evaluate existing models. Define how a list of candidate source texts, representative score ranges, and comparisons using a shared source should work, then determine how the standalone option could integrate with Clowder. Done means the requested scores can be generated consistently for each source.