facebookresearch / facebookresearch/detectron2

Machine crashing before run trainer.train() line

Open
#5,372 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

I ran a script to inference a custom dataset made by myself with more than 300 annotation following this tutorial: https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5 I used the following parameters in the config:

```
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ["objects" + "_" + "train"]
cfg.DATASETS.TEST = []
cfg.DATALOADER.NUM_WORKERS = 3
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 4
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.0025
cfg.SOLVER.MAX_ITER = 400
cfg.SOLVER.STEPS = []
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 200
cfg.MODEL.DEVICE = "cuda"
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.3
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
```

At the moment that I ran this line:
```
import numpy as np
from threading import Thread
from queue import Queue
import sys
import os
from tqdm import tqdm
import cv2 as cv
import torch
import json
import paho.mqtt.publish as publish
import time

np.set_printoptions(threshold=sys.maxsize)

# import some common detectron2 utilities
from detectron2 import model_zoo
from detectron2.config import get_cfg
from detectron2.engine import DefaultPredictor, DefaultTrainer
from detectron2.checkpoint import DetectionCheckpointer
# from detectron2.utils.video_visualizer import VideoVisualizer
from detectron2.utils.visualizer import ColorMode, Visualizer
from detectron2.modeling import build_model
from detectron2.data import MetadataCatalog, DatasetCatalog
from detectron2.structures import BoxMode, Boxes, Instances

from shapely.geometry import Polygon, Point
...
...
cfg.OUTPUT_DIR="./output"
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()
```

Unfortunately my execution didn't come to run the ```trainer.train()``` line and my machine crashed on the trainer.resume_or_load(resume=False), I checked my main memory and swap partitions with ```htop``` and at the time that my script came in this line the progress bar stays whole red and both bars went until their limit.

I ran the script directly as python .

I would like to know if anyone faced the same as me, and how to fix it!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.