google / google/qkeras

How to perform inference on an hardware with alpha='auto'

Open
#134 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
584
Forks
108
PR merge metrics
No merged PRs in 30d

Description

I'm trying to implement inference on a hardware using the Xilinx ap_fixed for a model quantized with alpha='auto'. With alpha=1 it is straightforward. The weights (after applying the quantizer) can be exported directly to the hardware. When alpha='auto' is more challenging. I have not found an explanation on how to compute the weights and the scale, so I have analyzed the code.
This is an extract of `quantized_bits` for alpha='auto':
```python
m = K.pow(2.0, K.cast_to_floatx(unsigned_bits))
m_i = K.pow(2.0, K.cast_to_floatx(self.integer))
x = x / m_i
levels = (2**(self.bits-1)-1) * 2 if self.symmetric else (2**self.bits)-1
scale = (K.max(abs(x), axis=axis, keepdims=True) * 2) / levels
v = tf.floor(tf.abs(x) / scale + 0.5)
mask = v < levels / 2
z = tf.sign(x) * tf.where(mask, v, tf.ones_like(v) * levels / 2)
xq = m_i * z / m
xq2 = scale * xq
```
My understanding is that `z` contains the integer representation of the weights that utilize the entire range of the type, that is the scale is optimal. `xq` are the floating point representation of `z`. and `xq2` the quantized weights in floating point representation that are actually used in the convolution during training. These can exceed the range of the type.

To implement this in the hardware I have to save `z` as the weights and compute `scale` which is a constant that have to be applied after the convolution. For alpha='po2' it would be the same but the scale can be applied as a bit shift.

If this is true, it would be nice to have a function that return `z` and `scale` as `quantized_bits` does not.
Thanks

Contributor guide

Open the contributing guide

Research direction

Start with the `quantized_bits` entry point and the `alpha='auto'` calculation shown in the issue. Verify how `z`, `scale`, and the convolution output map to Xilinx ap_fixed hardware, then define the returned values or documentation; done means the hardware export and post-convolution scaling are unambiguous.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
embedded-iot, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.