tensorflow / tensorflow/model-optimization

QAT support for LayerNormalization

Open
#942 0 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

feature request technique:qat
Dominant language
Python
Stars
1.6k
Forks
349
Avg merge
3d 2h
Merged PRs (30d)
1

Description

System information

  • TensorFlow version (you are using): 2.8
  • Are you willing to contribute it (Yes/No): Yes

Motivation
This would be beneficial for models that use this layer - this is for example used in Transformer models.

Describe the feature
Be able to run QAT on a model with the LayerNormalization layer.

Describe how existing APIs don't satisfy your use case (optional if obvious)
As an example, the following code snippet will fail:

import tensorflow as tf
import tensorflow_model_optimization as tfmot

from tensorflow import keras

model = keras.Sequential([
    keras.layers.InputLayer(input_shape=(28, 28)),
    keras.layers.Reshape(target_shape=(28, 28, 1)),
    keras.layers.Conv2D(filters=12, kernel_size=(3, 3), activation='relu'),
    keras.layers.LayerNormalization(axis=3),
    keras.layers.MaxPooling2D(pool_size=(2, 2)),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
])

quant_model = tfmot.quantization.keras.quantize_model(model)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.