Enable Video models for other tasks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 7.3k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 13
Description
🚀 The feature
It would be nice to enable video models to be used as encoders for other tasks then classification.
Motivation, pitch
It seems that video models (e.g. mvit, swin video..) are strictly connected internally to their classification head.
It could be useful to let use them in others tasks. Just to make an example we could use them as a multi-scale encoder to be connected to other components like an encoder-decoder architecture for video segmentation, 3d medical segmentation etc..
Alternatives
No response
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue points to torchvision/models/video; start by reviewing the video model implementations and how they are coupled to classification heads. Determine the interface needed for using these models as encoders, and consider how support would serve the proposed segmentation use cases. Done means video models can be used beyond classification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100