google / google/qkeras

Recurrent networks quantized_bits with alpha

Open
#73 1 comment 0 reactions 1 assignee Claimed by @lishanok View on GitHub
Dominant language
Python
Stars
584
Forks
108
PR merge metrics
No merged PRs in 30d

Description

`quantized_bits` should be the same using the default value `None` and `alpha = 1.0`.

`scale` set as `1.0` with `self.alpha = None`:
https://github.com/google/qkeras/blob/master/qkeras/quantizers.py#L550

`scale` set as `1.0` with `self.alpha = 1.0`:
https://github.com/google/qkeras/blob/master/qkeras/quantizers.py#L597

And can they return the same values as demonstrated with this example:

```
q_noalpha = quantized_bits(14, 4, 1)
q_alpha = quantized_bits(14, 4, 1, alpha=1.0)
testvalues = np.arange(-30,30,0.00001, dtype='float32')
np.argwhere((q_noalpha(testvalues) == q_alpha(testvalues)) == False)
```
Which returns an empty array meaning that both quantizers return the same values.

However with recurrent networks changing between these two changes the result. I have attached a small reproducible output that is based on the test code from `qrecurrent_test.py` to demonstrate the behavior. `QLSTM` and `QSimpleRNN` both give results that do not match.

```
np.random.seed(31)

tf.random.set_seed(31)

inputs = 2 * np.random.rand(10, 2, 4)

rnn = qkeras.QSimpleRNN

x = x_in = Input((2, 4), name='input')
x = rnn(
16,
activation=quantized_tanh(bits=8),
kernel_quantizer=quantized_bits(8, 0, 1, alpha=1.0),
#recurrent_quantizer=quantized_bits(8, 0, 1, alpha=1.0),
bias_quantizer=quantized_bits(8, 0, 1, alpha=1.0),
state_quantizer=quantized_bits(4, 0, 1, alpha=1.0),
name='qrnn_0')(
x)
x = qkeras.QDense(
4,
kernel_quantizer=quantized_bits(6, 2, 1, alpha=1.0),
bias_quantizer=quantized_bits(4, 0, 1),
name='dense')(
x)
x = Activation('softmax', name='softmax')(x)

model = Model(inputs=[x_in], outputs=[x])

# save weights
save_weights = model.get_weights()

original_output = model.predict(inputs).astype(np.float16)

x = x_in = Input((2, 4), name='input')
x = rnn(
16,
activation=quantized_tanh(bits=8),
kernel_quantizer=quantized_bits(8, 0, 1),
#recurrent_quantizer=quantized_bits(8, 0, 1),
bias_quantizer=quantized_bits(8, 0, 1),
state_quantizer=quantized_bits(4, 0, 1),
name='qrnn_0')(
x)
x = qkeras.QDense(
4,
kernel_quantizer=quantized_bits(6, 2, 1),
bias_quantizer=quantized_bits(4, 0, 1),
name='dense')(
x)
x = Activation('softmax', name='softmax')(x)

model = Model(inputs=[x_in], outputs=[x])

model.set_weights(save_weights)

output_no_alpha = model.predict(inputs).astype(np.float16)

print(original_output-output_no_alpha)

```

Expected output:
```
[[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]]
```

Actual output:

```
[[-0.02148 -0.0864 -0.01233 0.12036 ]
[ 0.02441 -0.03333 0.01453 -0.00537 ]
[ 0.08765 -0.0879 -0.003296 0.003418]
[ 0.02734 -0.1011 -0.006287 0.0801 ]
[-0.005615 -0.0481 0.03137 0.02222 ]
[ 0.004883 -0.0454 0.001862 0.03857 ]
[-0.04565 -0.03235 -0.00908 0.0874 ]
[-0.007324 -0.04956 -0.002075 0.05884 ]
[ 0.02588 -0.04425 0.01611 0.001953]
[ 0.02344 -0.04398 0.0426 -0.02197 ]]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.