facebookresearch / facebookresearch/detectron2

Fix for running panoptic segmentation on video in the Detectron2 Tutorial.ipynb

Open
#5,368 1 comment 1 reaction 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

## 📚 Documentation Issue

In the Detectron2 Tutorial.ipynb Google Colab notebook the segmentation on a video is broken.
[Detectron2 Tutorial.ipynb](https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5#scrollTo=cyA4VmKcF61k)

This issue can be fixed by installing another youtube-dl and setting the locale to UTF-8:
```
# Install dependencies, download the video, and crop 5 seconds for processing
import locale
locale.getpreferredencoding = lambda: "UTF-8"

#ERROR: Unable to extract uploader id
#!pip install youtube-dl
#!youtube-dl https://www.youtube.com/watch?v=ll8TgCZ0plk -f 22 -o video.mp4
!pip install https://github.com/yt-dlp/yt-dlp/archive/master.tar.gz
!yt-dlp https://www.youtube.com/watch?v=ll8TgCZ0plk -f 269 -o video.mp4
!ffmpeg -i video.mp4 -t 00:00:06 -c:v copy video-clip.mp4
```

The next code block returns an error as well, because of the relative paths used in the demo.py script:
```
# Run frame-by-frame inference demo on this video (takes 3-4 minutes) with the "demo.py" tool we provided in the repo.
#!git clone https://github.com/facebookresearch/detectron2
# Note: this is currently BROKEN due to missing codec. See https://github.com/facebookresearch/detectron2/issues/2901 for workaround.
#%run detectron2/demo/demo.py --config-file detectron2/configs/COCO-PanopticSegmentation/panoptic_fpn_R_101_3x.yaml --video-input video-clip.mp4 --confidence-threshold 0.6 --output video-output.mkv \
# --opts MODEL.WEIGHTS detectron2://COCO-PanopticSegmentation/panoptic_fpn_R_101_3x/139514519/model_final_cafdb1.pkl

#Fix:
#Set the path to the parent 'detectron2' folder
sys.path.insert(0, os.path.abspath('detectron2'))
#Open the detectron2 folder in the left side panel of Google Colab. In the demo folder you can find the demo.py script.
#Now you can modify the demo.py script. Comment or delete the import of VisualizationDemo, and add the following import:
#from predictor import VisualizationDemo
#When the pull request will be accepted, the manual modification of demo.py script will no longer be needed.

%run detectron2/demo/demo.py --config-file detectron2/configs/COCO-PanopticSegmentation/panoptic_fpn_R_101_3x.yaml --video-input video-clip.mp4 --confidence-threshold 0.6 --output video-output.mkv \
--opts MODEL.WEIGHTS detectron2://COCO-PanopticSegmentation/panoptic_fpn_R_101_3x/139514519/model_final_cafdb1.pkl
```

Contributor guide

Open the contributing guide

Research direction

Start with the Detectron2 Tutorial.ipynb video-segmentation cells and compare them with detectron2/demo/demo.py and the referenced predictor.py import. Run the notebook's video workflow in Colab, then verify that dependency installation, video download, frame-by-frame inference, and output generation complete without manual edits.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
computer-vision, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.