facebookresearch / facebookresearch/detectron2
Fix for running panoptic segmentation on video in the Detectron2 Tutorial.ipynb
- Dominant language
- Python
- Stars
- 34.7k
- Forks
- 7.9k
- PR merge metrics
- No merged PRs in 30d
Description
## 📚 Documentation Issue
In the Detectron2 Tutorial.ipynb Google Colab notebook the segmentation on a video is broken.
[Detectron2 Tutorial.ipynb](https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5#scrollTo=cyA4VmKcF61k)
This issue can be fixed by installing another youtube-dl and setting the locale to UTF-8:
```
# Install dependencies, download the video, and crop 5 seconds for processing
import locale
locale.getpreferredencoding = lambda: "UTF-8"
#ERROR: Unable to extract uploader id
#!pip install youtube-dl
#!youtube-dl https://www.youtube.com/watch?v=ll8TgCZ0plk -f 22 -o video.mp4
!pip install https://github.com/yt-dlp/yt-dlp/archive/master.tar.gz
!yt-dlp https://www.youtube.com/watch?v=ll8TgCZ0plk -f 269 -o video.mp4
!ffmpeg -i video.mp4 -t 00:00:06 -c:v copy video-clip.mp4
```
The next code block returns an error as well, because of the relative paths used in the demo.py script:
```
# Run frame-by-frame inference demo on this video (takes 3-4 minutes) with the "demo.py" tool we provided in the repo.
#!git clone https://github.com/facebookresearch/detectron2
# Note: this is currently BROKEN due to missing codec. See https://github.com/facebookresearch/detectron2/issues/2901 for workaround.
#%run detectron2/demo/demo.py --config-file detectron2/configs/COCO-PanopticSegmentation/panoptic_fpn_R_101_3x.yaml --video-input video-clip.mp4 --confidence-threshold 0.6 --output video-output.mkv \
# --opts MODEL.WEIGHTS detectron2://COCO-PanopticSegmentation/panoptic_fpn_R_101_3x/139514519/model_final_cafdb1.pkl
#Fix:
#Set the path to the parent 'detectron2' folder
sys.path.insert(0, os.path.abspath('detectron2'))
#Open the detectron2 folder in the left side panel of Google Colab. In the demo folder you can find the demo.py script.
#Now you can modify the demo.py script. Comment or delete the import of VisualizationDemo, and add the following import:
#from predictor import VisualizationDemo
#When the pull request will be accepted, the manual modification of demo.py script will no longer be needed.
%run detectron2/demo/demo.py --config-file detectron2/configs/COCO-PanopticSegmentation/panoptic_fpn_R_101_3x.yaml --video-input video-clip.mp4 --confidence-threshold 0.6 --output video-output.mkv \
--opts MODEL.WEIGHTS detectron2://COCO-PanopticSegmentation/panoptic_fpn_R_101_3x/139514519/model_final_cafdb1.pkl
```
Contributor guide
Research direction
Start with the Detectron2 Tutorial.ipynb video-segmentation cells and compare them with detectron2/demo/demo.py and the referenced predictor.py import. Run the notebook's video workflow in Colab, then verify that dependency installation, video download, frame-by-frame inference, and output generation complete without manual edits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- computer-vision, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100