Xilinx / Xilinx/Vitis-Tutorials
Unable to reprogram FPGA and run NEW kernels in combination with XRT, affects both non-DFX and DFX base platforms
Open
@imrickysu is already working on this.
Since Aug 18, 2021.
Working on it
- Dominant language
- C
- Stars
- 1.6k
- Forks
- 627
- PR merge metrics
- No merged PRs in 30d
Description
Context
This ticket's part of a debugging effort and connected to past issues including #69 #70 #71 and #72.
While developing kernels it's common to test (also in hw target) incremental variants of it. So far, unfortunately, I haven't been able to do so. Only the first kernel produced (the first time the sd card is introduced is the hardware) remains. I haven't been able to successfully change this. The current issues has been validated in the following hardware and base platform combinations:
| Hardware | Base platform |
|---|---|
| ZCU102 | xilinx_zcu102_base_202020_1 |
| ZCU102 | xilinx_zcu102_base_dfx_202020_1 |
| ZCU104 | xilinx_zcu104_base_202020_1 |
A simple way to reproduce this is to build and test variants (silly ones, simple modifications of the kernel) of https://github.com/Xilinx/Vitis-Tutorials/tree/master/Getting_Started/Vitis. An exemplary flow to reproduce this issue is as follows:
- Follow https://github.com/Xilinx/Vitis-Tutorials/blob/master/Getting_Started/Vitis/Part4.md and build the sd_card.img
- Flash raw image into SD card and boot
- Run
app.exeobtaining results coherent to thevadd.xclbinkernel (whatever they are) - Modify
vadd.cpp(source code) introduce a small (or big, up to you) change - Rebuild kernel with Vitis
v++compiler - Scp kernel to hardware as
vadd.xclbin.newfor DFX platforms (or just build the whole image complete again withv++ -pfor non-DFX platforms) - Run
app.exe vadd.xclbin.new
The expected observation is that result remains fixed to 3. and not related to the changes introduced in 4., 5. and 6.. A past issue in XRT identified a similar behavior https://github.com/Xilinx/XRT/issues/2747.
Results so far
NOTE: I'm using Vitis suite 2020.2 on top of Ubuntu 20.04.
NOTE 2: Ignore paths, they're specific to my development machines.
xilinx_zcu102_base_202020_1
Regardless of what I do the result's always the same which corresponds with the first kernel I flashed which modified https://github.com/Xilinx/Vitis-Tutorials/blob/master/Getting_Started/Vitis/example/src/vadd.cpp#L15 to:
out = in1 + 2 * in2
cd /home/xilinx/Vitis-Tutorials/Getting_Started/Vitis/example/zcu102/hw; rm -r package/ _x/ vadd.x* app.exe a.xclbin v++*
# configure env.
source /tools/Xilinx/Vitis/2020.2/settings64.sh
export PLATFORM_REPO_PATHS=/home/xilinx/xilinx_zcu102_base_202020_1
export ROOTFS=/home/xilinx/rootfs
source /home/xilinx/rootfs/sdk/environment-setup-aarch64-xilinx-linux
# build
${CXX} -Wall -g -std=c++11 ../../src/host.cpp -o app.exe -I/usr/include/xrt -lOpenCL -lpthread -lrt -lstdc++
v++ -c -t hw --config ../../src/zcu102.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xo
v++ -l -t hw --config ../../src/zcu102.cfg ./vadd.xo -o vadd.xclbin
v++ -p -t hw --config ../../src/zcu102.cfg ./vadd.xclbin --package.out_dir package --package.rootfs ${ROOTFS}/rootfs.ext4 --package.sd_file ${ROOTFS}/Image --package.sd_file xrt.ini --package.sd_file app.exe --package.sd_file vadd.xclbin --package.sd_file run_app.sh
# test acceleration
cd /media/sd-mmcblk0p1
export XILINX_XRT=/usr
./app.exe vadd.xclbin
root@zynqmp-common-2020_2:/media/sd-mmcblk0p1# ./app.exe vadd.xclbin
[ 265.853107] [drm] Pid 1221 opened device
[ 265.857065] [drm] Pid 1221 closed device
INFO: Found Xilinx Platform
[ 265.863503] [drm] Pid 1221 opened device
[ 265.869667] [drm] Pid 1221 closed device
[ 265.873637] [drm] Pid 1221 opened device
INFO: Loading 'vadd.xclbin'
[ 265.962660] [drm] zocl_xclbin_read_axlf The XCLBIN already loaded
[ 265.962676] [drm] zocl_xclbin_read_axlf c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 ret: 0
[ 265.971245] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 locked, ref=1
[ 265.978917] [drm] Reconfiguration not supported
[ 265.990677] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 unlocked, ref=0
[ 265.991808] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 locked, ref=1
[ 266.005109] [drm] User buffer is not physical contiguous
Error: Result mismatch
i = 0 CPU result = 2349 Device result = 4281
TEST FAILED
... (continues)
xilinx_zcu102_base_dfx_202020_1
cd /home/xilinx/Vitis-Tutorials/Getting_Started/Vitis/example/zcu102/hw; rm -r package/ _x/ vadd.x* app.exe a.xclbin v++*
# configure env.
source /tools/Xilinx/Vitis/2020.2/settings64.sh
export PLATFORM_REPO_PATHS=/home/xilinx/xilinx_zcu102_base_dfx_202020_1
export ROOTFS=/home/xilinx/rootfs
source /home/xilinx/rootfs/sdk/environment-setup-aarch64-xilinx-linux
# build
${CXX} -Wall -g -std=c++11 ../../src/host.cpp -o app.exe -I/usr/include/xrt -lOpenCL -lpthread -lrt -lstdc++
v++ -c -t hw --config ../../src/zcu102_dfx.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xo
v++ -l -t hw --config ../../src/zcu102_dfx.cfg ./vadd.xo -o vadd.xclbin
v++ -p -t hw --config ../../src/zcu102_dfx.cfg ./vadd.xclbin --package.out_dir package --package.rootfs ${ROOTFS}/rootfs.ext4 --package.sd_file ${ROOTFS}/Image --package.sd_file xrt.ini --package.sd_file app.exe --package.sd_file vadd.xclbin --package.sd_file run_app.sh
# test acceleration
cd /media/sd-mmcblk0p1
export XILINX_XRT=/usr
./app.exe vadd.xclbin
root@zynqmp-common-2020_2:/media/sd-mmcblk0p1# ./app.exe vadd.xclbin
[ 265.853107] [drm] Pid 1221 opened device
[ 265.857065] [drm] Pid 1221 closed device
INFO: Found Xilinx Platform
[ 265.863503] [drm] Pid 1221 opened device
[ 265.869667] [drm] Pid 1221 closed device
[ 265.873637] [drm] Pid 1221 opened device
INFO: Loading 'vadd.xclbin'
[ 265.962660] [drm] zocl_xclbin_read_axlf The XCLBIN already loaded
[ 265.962676] [drm] zocl_xclbin_read_axlf c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 ret: 0
[ 265.971245] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 locked, ref=1
[ 265.978917] [drm] Reconfiguration not supported
[ 265.990677] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 unlocked, ref=0
[ 265.991808] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 locked, ref=1
[ 266.005109] [drm] User buffer is not physical contiguous
Error: Result mismatch
i = 0 CPU result = 2349 Device result = 4281
TEST FAILED
(gets stucked in here)
xilinx_zcu104_base_202020_1
cd /home/xilinx/Vitis-Tutorials/Getting_Started/Vitis/example/zcu104/hw
# configure env.
source /tools/Xilinx/Vitis/2020.2/settings64.sh
export PLATFORM_REPO_PATHS=/home/xilinx/xilinx_zcu104_base_202020_1
# export ROOTFS=/home/xilinx/rootfs
export ROOTFS=/home/xilinx/petalinux-xilinx-zcu104-2020.2/images/linux
source /home/xilinx/rootfs/sdk/environment-setup-aarch64-xilinx-linux
# build
${CXX} -Wall -g -std=c++11 ../../src/host.cpp -o app.exe -I/usr/include/xrt -lOpenCL -lpthread -lrt -lstdc++
v++ -c -t hw --config ../../src/zcu104.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xo
v++ -l -t hw --config ../../src/zcu104.cfg ./vadd.xo -o vadd.xclbin
v++ -p -t hw --config ../../src/zcu104.cfg ./vadd.xclbin --package.out_dir package --package.rootfs ${ROOTFS}/rootfs.ext4 --package.sd_file ${ROOTFS}/Image --package.sd_file xrt.ini --package.sd_file app.exe --package.sd_file vadd.xclbin --package.sd_file run_app.sh
# run
...
TEST PASSED
I haven't been able to get tests to fail after this. Regardless of the modifications in the kernels.
Additional tests performed
Reinstalled Vitis suite and PetaLinux completely in a newly formatted machine with Ubuntu 20.04. Same result.
Temporary fix
A user proposed a temporary fix flashing manually the FPGA which I have not been able to reproduce https://github.com/Xilinx/XRT/issues/2747#issuecomment-583664143. Here're my results when attempting this:
xilinx_zcu104_base_202020_1
# in the workstation
xclbinutil --dump-section BITSTREAM:RAW:vadd.bit -i vadd.xclbin
# then scp-it to embedded
# in the embedded platform
cd /media/sd-mmcblk0p1
export XILINX_XRT=/usr
fpgautil -b ./vadd.bit
root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# ./app.exe
[ 158.257308] [drm] Pid 1028 opened device
[ 158.261267] [drm] Pid 1028 closed device
INFO: Found Xilinx Platform
[ 158.267677] [drm] Pid 1028 opened device
[ 158.273834] [drm] Pid 1028 closed device
[ 158.277794] [drm] Pid 1028 opened device
INFO: Loading 'vadd.xclbin'
[ 158.430057] [drm] zocl_xclbin_read_axlf The XCLBIN already loaded
[ 158.430073] [drm] zocl_xclbin_read_axlf 030235fc-c6ab-4195-83b3-00e2e0860b11 ret: 0
[ 158.438503] [drm] bitstream 030235fc-c6ab-4195-83b3-00e2e0860b11 locked, ref=1
[ 158.446209] [drm] Reconfiguration not supported
[ 158.457973] [drm] bitstream 030235fc-c6ab-4195-83b3-00e2e0860b11 unlocked, ref=0
(gets stucked in here)
Further debugging this issue led to a kernel panic:
root@xilinx-zcu104-2020_2:~# cd /media/sd-mmcblk0p1
root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# export XILINX_XRT=/usr
root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# fpgautil -b ./vadd.bit
[ 42.264337] fpga_manager fpga0: writing vadd.bit to Xilinx ZynqMP FPGA Manager
Time taken to load BIN is 9302.000000 Milli Seconds
BIN FILE loaded through FPGA manager successfully
root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1#
root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1#
root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# ls
BOOT.BIN boot.scr profile_summary.csv timeline_trace.csv xrt.ini
Image init.sh run_app.sh vadd.bit
app.exe platform_desc.txt system.dtb vadd.xclbin
root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# gdb ./app.exe
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-xilinx-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./app.exe...
(gdb) r
Starting program: /media/sd-mmcblk0p1/app.exe
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
[ 81.754011] Unable to handle kernel NULL pointer dereference at virtual address 00000000000007f3
[ 81.762787] Mem abort info:
[ 81.765563] ESR = 0x96000004
[ 81.768601] EC = 0x25: DABT (current EL), IL = 32 bits
[ 81.773895] SET = 0, FnV = 0
[ 81.776933] EA = 0, S1PTW = 0
[ 81.780057] Data abort info:
[ 81.782922] ISV = 0, ISS = 0x00000004
[ 81.786741] CM = 0, WnR = 0
[ 81.789695] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000877e81000
[ 81.796116] [00000000000007f3] pgd=0000000000000000
[ 81.800978] Internal error: Oops: 96000004 [#1] SMP
[ 81.805838] Modules linked in: dmaproxy(O) mali(O) zocl(O)
[ 81.811319] CPU: 0 PID: 977 Comm: gdb Tainted: G O 5.4.0-xilinx-v2020.2 #1
[ 81.819483] Hardware name: ZynqMP ZCU104 RevC (DT)
[ 81.824258] pstate: a0000085 (NzCv daIf -PAN -UAO)
[ 81.829039] pc : xilinx_dpdma_chan_err_task+0xe0/0x1b8
[ 81.834162] lr : xilinx_dpdma_chan_err_task+0x1c/0x1b8
[ 81.839281] sp : ffff800010003e40
[ 81.842580] x29: ffff800010003e40 x28: 0000000000000000
[ 81.847875] x27: 0000000000000038 x26: 0000000000000060
[ 81.853169] x25: 0000000000000040 x24: 00000000000000e0
[ 81.858464] x23: ffff00087a37f140 x22: ffff8000110c1b00
[ 81.863759] x21: 0000000000000000 x20: ffff000879678c80
[ 81.869054] x19: ffff0008796eae80 x18: 0000000000000000
[ 81.874348] x17: 0000000000000000 x16: 0000000000000000
[ 81.879643] x15: 0000000000000000 x14: 5f3874534e497373
[ 81.884938] x13: 656363615f4d5f39 x12: 617461645f796e41
[ 81.890233] x11: 5f3974534b4e5a5f x10: 0000000000000040
[ 81.895528] x9 : ffff8000110f5368 x8 : ffff8000110f5360
[ 81.900822] x7 : ffff00087f777858 x6 : ffff800010003dc0
[ 81.906117] x5 : ffff80086e6bf000 x4 : 0000000000000000
[ 81.911412] x3 : 0000000000000001 x2 : ffff00087682dcc8
[ 81.916707] x1 : ffff00087682dc80 x0 : 03000000000006f3
[ 81.922002] Call trace:
[ 81.924435] xilinx_dpdma_chan_err_task+0xe0/0x1b8
[ 81.929210] tasklet_action_common.isra.0+0xcc/0x178
[ 81.934164] tasklet_action+0x24/0x30
[ 81.937811] __do_softirq+0x118/0x22c
[ 81.941456] irq_exit+0x98/0xc0
[ 81.944581] __handle_domain_irq+0x64/0xb8
[ 81.948659] gic_handle_irq+0x5c/0xb8
[ 81.952305] el1_irq+0xb8/0x140
[ 81.955433] __arch_copy_to_user+0x1a0/0x218
[ 81.959694] copy_page_to_iter+0xe8/0x338
[ 81.963687] generic_file_read_iter+0x384/0xb28
[ 81.968210] ext4_file_read_iter+0x38/0x50
[ 81.972297] new_sync_read+0xe4/0x170
[ 81.975941] __vfs_read+0x2c/0x40
[ 81.979239] vfs_read+0xb8/0x190
[ 81.982451] ksys_read+0x68/0xf0
[ 81.985663] __arm64_sys_read+0x18/0x20
[ 81.989484] el0_svc_common.constprop.0+0x68/0x160
[ 81.994265] el0_svc_handler+0x6c/0x88
[ 81.997997] el0_svc+0x8/0xc
[ 82.000865] Code: f8448c40 eb00005f d1040000 540000a0 (f9408000)
[ 82.006952] ---[ end trace 1b932a3dbbd74e07 ]---
[ 82.011555] Kernel panic - not syncing: Fatal exception in interrupt
[ 82.017891] SMP: stopping secondary CPUs
[ 82.021799] Kernel Offset: disabled
[ 82.025269] CPU features: 0x0002,20002004
[ 82.029260] Memory Limit: none
[ 82.032302] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
xilinx_zcu102_base_dfx_202020_1
# in the workstation
xclbinutil --dump-section BITSTREAM:RAW:vadd.bit -i vadd.xclbin
# then scp-it to embedded
# in the embedded platform
cd /media/sd-mmcblk0p1
export XILINX_XRT=/usr
fpgautil -b ./vadd.bit
root@zynqmp-common-2020_2:/media/sd-mmcblk0p1# fpgautil -b ./vadd.bit
[ 430.136806] fpga_manager fpga0: writing vadd.bit to Xilinx ZynqMP FPGA Manager
[ 445.466900] mmc0: Timeout waiting for hardware interrupt.
[ 445.472292] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[ 445.478714] mmc0: sdhci: Sys addr: 0x00000000 | Version: 0x00001002
[ 445.485137] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[ 445.491560] mmc0: sdhci: Argument: 0x0042c800 | Trn mode: 0x00000027
[ 445.497983] mmc0: sdhci: Present: 0x01ff0000 | Host ctl: 0x0000001f
[ 445.504406] mmc0: sdhci: Power: 0x0000000f | Blk gap: 0x00000080
[ 445.510830] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000207
[ 445.517253] mmc0: sdhci: Timeout: 0x00000009 | Int stat: 0x00000002
[ 445.523676] mmc0: sdhci: Int enab: 0x03ff008b | Sig enab: 0x03ff008b
[ 445.530099] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000001
[ 445.536522] mmc0: sdhci: Caps: 0x35ecc881 | Caps_1: 0x00002007
[ 445.542945] mmc0: sdhci: Cmd: 0x0000193a | Max curr: 0x00000000
[ 445.549369] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x003b377f
[ 445.555792] mmc0: sdhci: Resp[2]: 0x325b5900 | Resp[3]: 0x00000c00
[ 445.562214] mmc0: sdhci: Host ctl2: 0x00000001
[ 445.566643] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x000000087926d2c0
[ 445.573759] mmc0: sdhci: ============================================
[ 456.306892] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 456.312799] rcu: 0-...0: (1 GPs behind) idle=926/0/0x1 softirq=4120/4121 fqs=2596
[ 456.320444] rcu: 1-...0: (1 GPs behind) idle=e82/1/0x4000000000000000 softirq=2437/2438 fqs=2596
[ 456.329391] (detected by 3, t=5257 jiffies, g=8457, q=24)
[ 456.334858] Task dump for CPU 0:
[ 456.338071] swapper/0 R running task 0 0 0 0x0000000a
[ 456.345109] Call trace:
[ 456.347548] __switch_to+0x1c4/0x288
[ 456.351108] 0xffff8000110d98a8
[ 456.354232] Task dump for CPU 1:
[ 456.357443] sh R running task 0 1257 1253 0x00000022
[ 456.364482] Call trace:
[ 456.366915] __switch_to+0x1c4/0x288
[ 456.370472] 0xfffffe0021cd95c0
(board hangs in here)
Questions
I was surprised that this (by "this" I mean the development flows explained above in Context) couldn't work in non-DFX platforms. Is this expected to work in non-DFX platforms? If not, what's the proposal for reprogramming the FPGA with XRT flows?addressed in https://github.com/Xilinx/Vitis-Tutorials/issues/74#issuecomment-838149806- I was very surprised that I couldn't get this flow working even with DFX base platform. Can you confirm this should work with DFX platforms?
Since none of the above worked and the proposed fix in a past issue only led to more issues, how can I manually reprogram FPGAs for using them with XRT?addressed in https://github.com/Xilinx/Vitis-Tutorials/issues/74#issuecomment-838149806
I'm currently blocked with this (and have been blocked by it for more than a week already) so It'd be great if someone could help with it or suggest and alternative route. Anything for ZCU102 or ZCU104 would do.
Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.