huggingface / huggingface/pytorch-image-models
[FEATURE] timm.models.adapt_input_conv: beyond RGB weights
- Dominant language
- Python
- Stars
- 37.1k
- Forks
- 5.2k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 34
Description
**Is your feature request related to a problem? Please describe.**
TorchGeo provides a number of model weights pre-trained on non-RGB imagery (e.g., Sentinel-2, 13 channels). Oftentimes, when dealing with time-series data, we would like to stack images along the channel dimension so that we end up with $$B \times TC \times H \times W$$ inputs. However, we don't yet have an easy way to adapt our pre-trained weights to match.
**Describe the solution you'd like**
`timm.models.adapt_input_conv` provides a powerful tool for repeating and scaling weights to adapt to changing `in_chans`, but only seems to support 3-channel weights if `in_chans` > 1. I would like to extend this to support any number of channels. Would this be as simple as replacing 3 with `I` throughout the function?
**Describe alternatives you've considered**
We could write our own functionality in TorchGeo, but figured this would be useful to the broader timm community.
**Additional context**
@isaaccorley @keves1 may also be interested in this.
Contributor guide
Research direction
Start at the timm.models.adapt_input_conv function mentioned in the issue and inspect how it repeats and scales pretrained input weights for different in_chans. Extend the behavior beyond 3-channel weights while preserving existing adaptation behavior, then verify that arbitrary channel counts such as Sentinel-2 inputs and stacked time-series channels are supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100