Ability to import CocoEvaluator

🚀 Feature

It would be great if we could do the following:

from torchvision import CocoEvaluator

base_ds = dataset.coco
iou_types = ['bbox']
coco_evaluator = CocoEvaluator(base_ds, iou_types) # initialize evaluator with ground truths

model = ... # fancy deep learning model
model.eval()

print("Running evaluation...")
for batch in data_loader:
    outputs = model(**batch)
    result = ... # convert outputs of model to COCO api
    coco_evaluator.update(result)

coco_evaluator.synchronize_between_processes()
coco_evaluator.accumulate()
coco_evaluator.summarize()

Right now, CocoEvaluator cannot be imported as it's currently under the references directory in this repository, it's not part of the torchvision package.

Motivation

I'm currently implementing DETR (end-to-end object detection with Transformers), and right now I have to copy all of this code of COCO evaluation in order to evaluate the model. The authors of DETR also copied a lot of the code for evaluation into their own repository. It would be great if we can simply import it, and run evaluation of a deep learning model.

Even in the official torchvision tutorial, they state that:

"In references/detection/, we have a number of helper functions to simplify training and evaluating detection models. Here, we will use references/detection/engine.py, references/detection/utils.py and references/detection/transforms.py. Just copy everything under references/detection to your folder and use them here."
=> life would be easier if users don't need to look into Github repos and copy files into their own folder. Also, there would be a central place (namely this repository) where the official COCO evaluation is defined, and can be updated in the future. Right now evaluation is cluttered across hundreds of Github repos.

This would also foster reproducability of experiments with object detection models, as right now it's a lot of work to just evaluate a model with metrics like mAP.

Start in the references/detection code linked by the issue and inspect where CocoEvaluator is defined and how the official torchvision tutorial uses it. Determine the package entry point and API surface needed for from torchvision import CocoEvaluator; done means the evaluator is importable from the package and its documented COCO evaluation workflow remains usable.