facebookresearch / facebookresearch/detectron2

"No predictions from the model" warning during Colab notebook evaluation

Open
#4,891 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

I tried to follow the template as directed, but please let me know if something is missing (long-time reader, first-time poster). I have also added a link to view the Colab notebook: https://colab.research.google.com/drive/1UIcCMZBpE7AVjv4RhRuXy0ECd4BL6ssH?usp=sharing

## Instructions To Reproduce the Issue:

After my model has ostensibly trained successfully, I try to evaluate the model using AP metric in COCO API per the Detectron2 Colab notebook instructions. This is the script I use:
```
from detectron2.evaluation import COCOEvaluator, inference_on_dataset
from detectron2.data import build_detection_test_loader
evaluator = COCOEvaluator("handwriting_val", output_dir="./output")
val_loader = build_detection_test_loader(cfg, "handwriting_val")
print(inference_on_dataset(predictor.model, val_loader, evaluator))
```

But when I receive following message when the script has run:
```
[03/30 19:22:29 d2.evaluation.coco_evaluation]: Fast COCO eval is not built. Falling back to official COCO eval.
WARNING [03/30 19:22:29 d2.data.datasets.coco]:
Category ids in annotations are not in [1, #categories]! We'll apply a mapping for you.

[03/30 19:22:29 d2.data.datasets.coco]: Loaded 10 images in COCO format from /content/gdrive/MyDrive/odm_coco/valid/_annotations.coco.json
[03/30 19:22:29 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [ResizeShortestEdge(short_edge_length=(800, 800), max_size=1333, sample_style='choice')]
[03/30 19:22:29 d2.data.common]: Serializing the dataset using:
[03/30 19:22:29 d2.data.common]: Serializing 10 elements to byte tensors and concatenating them all ...
[03/30 19:22:29 d2.data.common]: Serialized dataset takes 0.00 MiB
[03/30 19:22:29 d2.evaluation.evaluator]: Start inference on 10 batches
/usr/local/lib/python3.9/dist-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
[03/30 19:22:31 d2.evaluation.evaluator]: Total inference time: 0:00:00.590660 (0.118132 s / iter per device, on 1 devices)
[03/30 19:22:31 d2.evaluation.evaluator]: Total inference pure compute time: 0:00:00 (0.061980 s / iter per device, on 1 devices)
[03/30 19:22:31 d2.evaluation.coco_evaluation]: Preparing results for COCO format ...
[03/30 19:22:31 d2.evaluation.coco_evaluation]: Saving results to ./output/coco_instances_results.json
[03/30 19:22:31 d2.evaluation.coco_evaluation]: Evaluating predictions with official COCO API...
WARNING [03/30 19:22:31 d2.evaluation.coco_evaluation]: No predictions from the model!
OrderedDict([('bbox', {'AP': nan, 'AP50': nan, 'AP75': nan, 'APs': nan, 'APm': nan, 'APl': nan})])
```

**1. Full runnable code or full changes you made:**

After installing Detectron2 and all its requirements per the Google Colab notebook, I register my COCO dataset w/ this script:
from detectron2.data.datasets import register_coco_instances:

```
import os
import numpy as np
import json
from detectron2.structures import BoxMode
from detectron2.data import DatasetCatalog, MetadataCatalog

register_coco_instances("handwriting_train", {}, "/content/gdrive/MyDrive/odm_coco/train/_annotations.coco.json", "/content/gdrive/MyDrive/odm_coco/train")
register_coco_instances("handwriting_val", {}, "/content/gdrive/MyDrive/odm_coco/valid/_annotations.coco.json", "/content/gdrive/MyDrive/odm_coco/valid")

handwriting_train_metadata = MetadataCatalog.get("handwriting_train")
handwriting_test_metadata = MetadataCatalog.get("handwriting_val")
```

After which, I confirm it the data is in the correct format w/ this:
```
import random
from detectron2.utils.visualizer import Visualizer

handwriting_train_metadata = MetadataCatalog.get("handwriting_train")
dataset_dicts = DatasetCatalog.get("handwriting_train")

for d in random.sample(dataset_dicts, 3):
img = cv2.imread(d["file_name"])
visualizer = Visualizer(img[:, :, ::-1], metadata=handwriting_train_metadata, scale=0.5)
vis = visualizer.draw_dataset_dict(d)
cv2_imshow(vis.get_image()[:, :, ::-1])
```

My script for training the model is as follows:
```
from detectron2.engine import DefaultTrainer

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("handwriting_train",)
cfg.DATASETS.TEST = ("handwriting_val",)
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 300
cfg.SOLVER.STEPS = []
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.1

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()
```
After this, I run the script provided by the Colab notebook to create a predictor from the model

```
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.1
predictor = DefaultPredictor(cfg)
```

Then I run the script mentioned first above to get the evaluation results, but I receive the message that there are no predictions, as pasted above. Similarly, when I run the Colab notebook's script to visualize the prediction results, a sample of my annotated images are displayed, but without bounding boxes or anything similar on the images to indicate the model's predictions. This latter script is as follows:

```
from detectron2.utils.visualizer import ColorMode

dataset_dicts = DatasetCatalog.get("handwriting_val")
for d in random.sample(dataset_dicts, 3):
img = cv2.imread(d["file_name"])
outputs = predictor(img)
v = Visualizer(img[:, :, ::-1],
metadata=handwriting_train_metadata,
scale=0.5,
instance_mode=ColorMode.IMAGE_BW
)
out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
cv2_imshow(out.get_image()[:, :, ::-1])
```
As you'll see, I tried lower the threshold in both the evaluation and training scripts, to troubleshoot the error, but this has not appeared to alleviate the issue. Any help/insight is appreciated!

## Expected behavior:

Return test metric values and prediction results.

## Environment:

I've copied and pasted the provided code exactly:
```
wget -nc -nv https://github.com/facebookresearch/detectron2/raw/main/detectron2/utils/collect_env.py && python collect_env.py
```
But all I receive in the Colab notebook is an error claiming "invalid syntax."

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.