mapbox / mapbox/robosat

Allow arbitrary number of input channels in ResNet encoder (not only RGB)

Open
#56 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.1k
Forks
385
PR merge metrics
No merged PRs in 30d

Description

With https://github.com/mapbox/robosat/pull/46 we are changing our model architecture from training the encoder and decoder from scratch to using a pre-trained ResNet for the encoder. The pre-trained ResNet uses three channels (RGB) for the input layer through.

We need to be able to add arbitrary channels, say, RGB + water mask + elevation + lidar. To to this we need to construct a wrapper module architecture extending the ResNet architecture, copying weights over, and initializing the new channels with zero. In addition the channel-wise mean and std dev needs to be adapted.

Tasks

  • Figure out how to extend the ResNet input channels
  • Figure out how to copy over the pre-trained ResNet parts
  • Let users construct a model with arbitrary channels
  • Adapt mean and std dev

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the ResNet encoder construction and the code that applies channel-wise mean and standard deviation; the issue does not name specific files or tests. Review how the existing three-channel model is built and determine how arbitrary input channels, pretrained weights, and normalization should be represented. Done means users can construct the encoder with additional channels while preserving pretrained RGB behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.