automl / automl/ConfigurableOptimizer

Changing the number of channels in a convolution is broken

Open
#27 0 comments 0 reactions 1 assignee Assigned to @shakibamrd View on GitHub
Medium priority old
Dominant language
Python
Stars
12
Forks
2
PR merge metrics
No merged PRs in 30d

Description

The code [here](https://github.com/automl/ConfigurableOptimizer/blob/1378cbb1b20df95f75cd1457c8308b160421fbb1/src/confopt/utils/reduce_channels.py#L52) in `def increase_conv_channels` instantiates a new convolution with more filters. My concerns here are twofold:

1. The new convolution is randomly initialized. We do not inherit anything from the original convolution. This is not the desired behavior
2. If a new convolution is instantiated, how is the optimizer made aware of its parameters?
Let's say that initially there was convolution _C1_ with parameters _C1_theta_ . The optimizer is initialized with _C1_theta_. Later, a new convolution _C2_ with new parameters _C2_theta_ is instantiated, replacing _C1_ in the supernet. The model would indeed consume ~_C1_~ _C2_ in its forward pass, but the optimizer is not aware of _C2_theta_.

Similarly, [this line](https://github.com/automl/ConfigurableOptimizer/blob/1378cbb1b20df95f75cd1457c8308b160421fbb1/src/confopt/utils/reduce_channels.py#L120) in `def reduce_conv_channels` also instantiates a new model. But here, the values from the original convolution are _copied_ onto the new convolution (with `weight.data`). This still doesn't work because the optimizer is not aware of the new convolution and its parameters.

EDIT: corrected mistake in the explanation in 2.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.