huggingface / huggingface/pytorch-image-models

[FEATURE] timm.models.adapt_input_conv: beyond RGB weights

Open
#2,445 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
37.1k
Forks
5.2k
Avg merge
1d 13h
Merged PRs (30d)
34

Description

**Is your feature request related to a problem? Please describe.**

TorchGeo provides a number of model weights pre-trained on non-RGB imagery (e.g., Sentinel-2, 13 channels). Oftentimes, when dealing with time-series data, we would like to stack images along the channel dimension so that we end up with $$B \times TC \times H \times W$$ inputs. However, we don't yet have an easy way to adapt our pre-trained weights to match.

**Describe the solution you'd like**

`timm.models.adapt_input_conv` provides a powerful tool for repeating and scaling weights to adapt to changing `in_chans`, but only seems to support 3-channel weights if `in_chans` > 1. I would like to extend this to support any number of channels. Would this be as simple as replacing 3 with `I` throughout the function?

**Describe alternatives you've considered**

We could write our own functionality in TorchGeo, but figured this would be useful to the broader timm community.

**Additional context**

@isaaccorley @keves1 may also be interested in this.

Contributor guide

Open the contributing guide

Research direction

Start at the timm.models.adapt_input_conv function mentioned in the issue and inspect how it repeats and scales pretrained input weights for different in_chans. Extend the behavior beyond 3-channel weights while preserving existing adaptation behavior, then verify that arbitrary channel counts such as Sentinel-2 inputs and stacked time-series channels are supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.