Problems with inference in PYNQ-Z1 and emulation stitched IP

Versions

• PYNQ Z1: v3.0.1
• FINN: v0.9
• Xilinx tools: 2022.2
• Ununtu: 20.04

Commit hash

commit e76f20d1d8d05f2d8ddb52ade0f991915672622b (HEAD -> dev, origin/dev)
Merge: a3b6a7fb 3873325a
Author: auphelia 56755897+auphelia@users.noreply.github.com
Date: Tue Jul 11 10:21:26 2023 +0100

Merge pull request #852 from Xilinx/fix/alveo_build

Set axilite address range to a minimum of 4K

commit 3873325a31897b8ccbde9a211f90d5184338368e
Author: auphelia jakobapk@web.de
Date: Tue Jul 11 09:44:30 2023 +0100

[AlveoBuild] Set axilite address range to a minimum of 4K

commit a3b6a7fbbc70224571656242eff57bc452f6753f
Merge: e56e8136 96fc4f57
Author: auphelia 56755897+auphelia@users.noreply.github.com
Date: Mon Jul 10 09:14:01 2023 +0100

Merge pull request #844 from Xilinx/feature/2022_2

Dev PR to update Docker environment to Ubuntu 22, Python 3.10 and Xilinx tool version

commit 96fc4f57670811fafe1753a63bf0ccfc521da077
Author: auphelia jakobapk@web.de
Date: Fri Jul 7 15:54:13 2023 +0100

[Deps] Update qonnx version

commit 7924bf7271b41dd808feac0e8c5017222490f553
Author: auphelia jakobapk@web.de
Date: Fri Jul 7 14:31:14 2023 +0100

[NBs] Update notebooks to only use QONNX export

commit 391cd76ee3edb6e802d9b565a99993c775cc2194
Author: auphelia jakobapk@web.de
Date: Fri Jul 7 12:07:42 2023 +0100

[deps] Bump clize to 5.0.1 and sigtools to 4.0.1

commit a48b5037871468e8a3e890b4719258c7dd1736e2
Author: auphelia jakobapk@web.de
Date: Thu Jul 6 16:50:29 2023 +0100

[Tests] Update tests to only use qonnx export

commit 0cd757fbdabea18779f5374842b45a4fd755db10
Author: auphelia jakobapk@web.de
Date: Thu Jul 6 15:50:01 2023 +0100

Quick summary

I am trying to implement the Lenet5 network on the PYNQ-Z1 board. For that purpose I have created the network using brevitas and I have obtained the following accuracy after training (about 55%).

I have followed all the steps of the FINN end-to-end flow and even all the intermediate checks (including emulation via PyVerilator).

At first I thought that all the intermediate checks were working correctly and I performed the deployment on the PYNQ board getting only 8% accuracy well below the 55% obtained with brevitas.

But the other day I realised that when performing the stitched IP emulation I always get the same output value, regardless of the input value.

Details

I'm using the next dataset: https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz

Steps to Reproduce

Add what needs to be done to reproduce the bug. Add code examples where useful
and make sure to include the resulting ONNX files, and the commit hash you are working on.

1. I create the lenet network in brevitas ( Note that I am using QuantIdentity with a bit width of 8 at the beginning and I am using biasing, except in the last layer I am not using biasing to avoid problems in the subsequent transformations to HLS layers)

BIT_WIDTH=2;

class QuantWeightActBiasLeNet(Module):
    def __init__(self):
        super(QuantWeightActBiasLeNet, self).__init__()
        self.quant_inp = qnn.QuantIdentity(bit_width=8, return_quant_tensor=True)
        self.conv1 = qnn.QuantConv2d(3, 6, 5, bias=True, weight_bit_width=BIT_WIDTH)
        self.relu1 = qnn.QuantReLU(bit_width=BIT_WIDTH, return_quant_tensor=True)
        self.conv2 = qnn.QuantConv2d(6, 16, 5, bias=True, weight_bit_width=BIT_WIDTH)
        self.relu2 = qnn.QuantReLU(bit_width=BIT_WIDTH, return_quant_tensor=True)
        self.fc1   = qnn.QuantLinear(16*5*5, 120, bias=True, weight_bit_width=BIT_WIDTH)
        self.relu3 = qnn.QuantReLU(bit_width=BIT_WIDTH, return_quant_tensor=True)
        self.fc2   = qnn.QuantLinear(120, 84, bias=True, weight_bit_width=BIT_WIDTH)
        self.relu4 = qnn.QuantReLU(bit_width=BIT_WIDTH, return_quant_tensor=True)
        self.fc3   = qnn.QuantLinear(84, 5, bias=False, weight_bit_width=BIT_WIDTH)

    def forward(self, x):
        out = self.quant_inp(x)
        out = self.relu1(self.conv1(out))
        out = F.max_pool2d(out, 2)
        out = self.relu2(self.conv2(out))
        out = F.max_pool2d(out, 2)
        out = torch.flatten(out,1)
        out = self.relu3(self.fc1(out))
        out = self.relu4(self.fc2(out))
        out = self.fc3(out)
       
        return out

2. Network training

3. Brevitas export

ready_model_filename = "Lenet_quantized.onnx"
export_qonnx(model,torch.randn(1,3,32,32), ready_model_filename)
qonnx_cleanup(ready_model_filename, out_file=ready_model_filename)

4. Tidy up, pre and post processing.

5. Lowering and streamlined transformations

model = ModelWrapper("lenet_quantized_pre_post.onnx")
model = model.transform(MoveScalarLinearPastInvariants())
model = model.transform(Streamline())
model = model.transform(LowerConvsToMatMul())
model = model.transform(MakeMaxPoolNHWC())
model = model.transform(absorb.AbsorbTransposeIntoMultiThreshold())

model = model.transform(MakeMaxPoolNHWC())
model = model.transform(absorb.AbsorbConsecutiveTransposes())

model = model.transform(Streamline())

model = model.transform(absorb.AbsorbScalarMulAddIntoTopK())
model = model.transform(InferDataLayouts())
model = model.transform(RemoveUnusedTensors())
model.save("lenet_quantized_streamlined.onnx")

6.Conversion to HLS layers

mem_mode = "decoupled"

model = model.transform(to_hls.InferBinaryMatrixVectorActivation(mem_mode))
model = model.transform(to_hls.InferQuantizedMatrixVectorActivation(mem_mode))

model = model.transform(to_hls.InferLabelSelectLayer())
model = model.transform(to_hls.InferThresholdingLayer())
model = model.transform(GiveUniqueNodeNames())

model = model.transform(to_hls.InferThresholdingLayer())
model = model.transform(to_hls.InferConvInpGen())
model = model.transform(to_hls.InferStreamingMaxPool())

model = model.transform(RemoveCNVtoFCFlatten())

model = model.transform(absorb.AbsorbConsecutiveTransposes())

model = model.transform(InferDataLayouts())

model.save("lenet_hls_layers.onnx")

7.Dataflow partitioning

8.Folding

model = ModelWrapper("flowers_dataflow_model.onnx")
fc_layers = model.get_nodes_by_op_type("MatrixVectorActivation")
# each tuple is (PE, SIMD, in_fifo_depth) for a layer
folding = [
    (6, 3),
    (2, 6),
    (2, 2),
    (1, 1),
    (1, 1),
  
]
for fcl, (pe, simd) in zip(fc_layers, folding):
    fcl_inst = getCustomOp(fcl)
    fcl_inst.set_nodeattr("PE", pe)
    fcl_inst.set_nodeattr("SIMD", simd)
    

# use same SIMD values for the sliding window operators
swg_layers = model.get_nodes_by_op_type("ConvolutionInputGenerator")
for i in range(len(swg_layers)):
    swg_inst = getCustomOp(swg_layers[i])
    simd = folding[i][1]
    swg_inst.set_nodeattr("SIMD", simd)
    

model = model.transform(GiveUniqueNodeNames())
model.save("flowers_lenet_folded.onnx")

9. Simulation cppsim: works correctly

10. Emulation node by node PyVerilator: works correctly

11. Emulation stitched IP PyVerilator: PROBLEM: always get the same output value, regardless of the input value

12. Deployment on PYNQ: PROBLEM: 8% inference accuracy