Raspberry Pi Zero W: LLVM optimization causes "unsupported FP instruction in kernel mode"

Open
#5,877 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
c

Research direction

Start with arch/arm/vfp/vfpmodule.c, especially VFP_bounce, and compare the LLVM-built disassembly with the GCC or patched-kernel behavior described here. Reproduce with ARCH=arm CC=clang LLVM=1 on a Raspberry Pi Zero W and run the linked test program. Done means the kernel no longer panics with an unsupported FP instruction when the test runs.

Written by the indexing model from the issue text.

Description

Describe the bug

On Raspberry Pi Zero W, If a kernel is built with LLVM, using vfp causes an "unsupported FP instruction in kernel mode" exception and kernel panic.
panic

Steps to reproduce the behaviour
  • Clone repo: git clone --depth=1 -b rpi-6.6.y https://github.com/raspberrypi/linux/
  • To build with llvm, patch some files(not related to this issue): curl https://github.com/raspberrypi/linux/commit/f43a67d9309a988eb1def20bc422c0882ce65e5a.patch | patch -p1
  • Build kernel with LLVM: make ARCH=arm CC=clang LLVM=1
  • Boot into a new kernel
  • Compile and run test program from https://github.com/raspberrypi/linux/issues/859#issuecomment-84036262
  • Kernel crashes
Device (s)

Raspberry Pi Zero W / WH

System

Kernel: 6.6.12+(7df1d3b44368209517fc7dc0b35ee2621a94722e)
Compiler: clang version 16.0.6 / clang version 17.0.6
Distribution: Gentoo Linux

Logs

No response

Additional context

The stack trace above shows an "undefined instruction" occurs at vfp_support_entry + 0xd4.
I disassembled the problematic vfp_support_entry dump.txt
vfp_support_entry + 0xd4 = 0x1540:

0000146c <vfp_support_entry>:
...
    1518:	e317020a 	tst	r7, #-1610612736	@ 0xa0000000
    151c:	1a000005 	bne	1538 <vfp_support_entry+0xcc>
    1520:	eef10a10 	vmrs	r0, fpscr
    1524:	e3100a01 	tst	r0, #4096	@ 0x1000
    1528:	1a000002 	bne	1538 <vfp_support_entry+0xcc>
    152c:	e3100807 	tst	r0, #458752	@ 0x70000
    1530:	0a000061 	beq	16bc <vfp_support_entry+0x250>
    1534:	e3877202 	orr	r7, r7, #536870912	@ 0x20000000
    1538:	e594003c 	ldr	r0, [r4, #60]	@ 0x3c
    153c:	e3a09010 	mov	r9, #16
    1540:	eef16a10 	vmrs	r6, fpscr
    1544:	e389940f 	orr	r9, r9, #251658240	@ 0xf000000
    1548:	e2800004 	add	r0, r0, #4
    154c:	e584003c 	str	r0, [r4, #60]	@ 0x3c
    1550:	e3e0009f 	mvn	r0, #159	@ 0x9f
    1554:	e240032e 	sub	r0, r0, #-1207959552	@ 0xb8000000
    1558:	e0070000 	and	r0, r7, r0
    155c:	eee80a10 	vmsr	fpexc, r0
    1560:	eef00a10 	vmrs	r0, fpsid
...

This part is originated from inlined VFP_bounce https://github.com/raspberrypi/linux/blob/7df1d3b44368209517fc7dc0b35ee2621a94722e/arch/arm/vfp/vfpmodule.c#L324-L350
In VFP_bounce, there are three fmxr/fmrx instructions in this order: fmxr(FPEXC, ...), fmrx(FPSID), fmrx(FPSCR).
But as we can see in dump.txt, LLVM reorders them into fmrx(FPSCR)(0x1540), fmxr(FPEXC, ...)(0x155c), fmrx(FPSID)(0x1560) and this is the cause of panic.

I tried to prevent llvm from reordering these instructions and found that removing static from VFP_bounce signature works.
Here is a disassemble snippet of the patched kernel: dump-patched.txt
With this fix, the test program runs without kernel panic.

Dominant language
C
Stars
13.2k
Forks
5.5k
Avg merge
2d 21h
Merged PRs (30d)
21

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from raspberrypi/linux

All issues in raspberrypi/linux

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.