Why do tasks include evaluation criteria?

@hildeweerts pointed out to me that some of the tasks in CC18 were replaced because of the evaluation criterion. We already ran some experiments, which will make it harder to aggregate results, but actually it has no real impact.
Which begs the question: why is the evaluation criterion even included in the task? Why would someone not compare across runs that are on the same dataset with the same target and same cross-validation routine?
Right now I expect that doesn't happen that much, because creating tasks is complicated (via the website or python interface) so no-one does it. But I don't understand this design decision given that we compute multiple metrics for each task.

Shouldn't the metrics be associated with task types and not tasks?

Start by reviewing task creation through the website and Python interface, then trace how evaluation criteria and metrics are associated with tasks and task types. Done means reaching a documented design decision about whether metrics should belong to task types, with the impact on existing experiments and result aggregation understood.