Allow arbitrary number of input channels in ResNet encoder (not only RGB)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 385
- PR merge metrics
- No merged PRs in 30d
Description
With https://github.com/mapbox/robosat/pull/46 we are changing our model architecture from training the encoder and decoder from scratch to using a pre-trained ResNet for the encoder. The pre-trained ResNet uses three channels (RGB) for the input layer through.
We need to be able to add arbitrary channels, say, RGB + water mask + elevation + lidar. To to this we need to construct a wrapper module architecture extending the ResNet architecture, copying weights over, and initializing the new channels with zero. In addition the channel-wise mean and std dev needs to be adapted.
Tasks
- Figure out how to extend the ResNet input channels
- Figure out how to copy over the pre-trained ResNet parts
- Let users construct a model with arbitrary channels
- Adapt mean and std dev
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the ResNet encoder construction and the code that applies channel-wise mean and standard deviation; the issue does not name specific files or tests. Review how the existing three-channel model is built and determine how arbitrary input channels, pretrained weights, and normalization should be represented. Done means users can construct the encoder with additional channels while preserving pretrained RGB behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100