Auto-Fuse ReLU with BatchNorm
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
When constructing my model in `tf.keras`, I have a repeated sequence of layers that goes:
```
tf.keras.layers.Conv2D(...)
tf.keras.layers.BatchNormalization(...)
tf.keras.layers.ReLU(...)
```
Due to the BatchNorm layer in between the convolution and activation function, I cannot specify a` Fused ReLU `directly within the `Conv2D` layer. Therefore, I lose out on reaping the benefits of `Fused ReLU`.
When converting the model to CoreML, `BatchNorm` layers are automatically fused in so we are just left with: `Conv2D -> ReLU` after model conversion.
I would like to see coremltools **automatically** fuse ReLU into the `Fused Conv2D+BN` layer (currently, when inspecting the model with Netron, I see that "Convolution" and "activation' are separate layers).
This would be pretty convenient for a lot of ML practitioners looking to optimize their models.
Contributor guide
Research direction
Start by tracing the CoreML conversion of the tf.keras Conv2D-BatchNormalization-ReLU sequence and compare the converted model in Netron. Done means the converted model represents the convolution, batch normalization, and ReLU as a fused layer rather than separate Convolution and activation layers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100