Kernel panic during uart overrun

Open
#3,226 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
c, linux

Research direction

Start with the uart_throttle entry in the reported trace, then follow tty_throttle_safe and the n_tty_receive_buf_common path. Reproduce the PL011 overrun at 1000000 baud with delayed userspace reads and determine why it reaches a null function pointer. Done means the workload no longer panics and the overrun is reported through the hardware counters.

Written by the indexing model from the issue text.

Description

Description
I am seeing a kernel panic consistently when operating the serial port at 1000000 baud and with no flow control using /dev/ttyAMA0 (PL011 device), with a data usage of greater 60%.
As far as I can tell it occurs when I do not read the data from user space quickly enough.

Expected behaviour
I would expected data to be dropped and hardware counters to show this overrun. eg
cat /proc/tty/driver/ttyAMA
0: uart:PL011 rev2 mmio:0x3F201000 irq:81 tx:0 rx:121501502 brk:4 oe:15 bo:119400638 RTS|CTS|DTR

System
I have seen this happening on a raspberry Pi2B and Pi3B+

cat /etc/rpi-issue

Raspberry Pi reference 2019-07-10
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 175dfb027ffabd4b8d5080097af0e51ed9a4a56c, stage2

vcgencmd version

Aug 15 2019 12:06:42 
Copyright (c) 2012 Broadcom
version 0e6daa5106dd4164474616408e0dc24f997ffcf3 (clean) (release) (start)

uname -a

Linux raspberrypi 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux

43.508528] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[ 43.512488] pgd = 4bef5981
[ 43.514451] [00000000] *pgd=00000000
[ 43.516421] Internal error: Oops: 80000007 [#1] SMP ARM
[ 43.518405] Modules linked in: sha256_generic cfg80211 rfkill 8021q garp stp llc bcm2835_codec(C) snd_bcm2835(C) bcm2835_v4l2(C) v4l2_mem2mem snd_pcm bcm2835_mmal_vchiq(C) videobuf2_dma_contig raspberrypi_hwmon v4l2_common videobuf2_vmalloc hwmon videobuf2_memops videobuf2_v4l2 snd_timer videobuf2_common snd videodev media vc_sm_cma(C) uio_pdrv_genirq uio fixed ledtrig_netdev ip_tables x_tables ipv6
[ 43.526397] CPU: 0 PID: 44 Comm: kworker/u8:1 Tainted: G C 4.19.66-v7+ #1253
[ 43.528616] Hardware name: BCM2835
[ 43.529741] Workqueue: events_unbound flush_to_ldisc
[ 43.530882] PC is at (null)
[ 43.532026] LR is at uart_throttle+0x118/0x124
[ 43.533158] pc : [<00000000>] lr : [<805870ac>] psr: 20000013
[ 43.534306] sp : b9fb3de0 ip : 00000001 fp : b9fb3dfc
[ 43.535442] r10: b764f200 r9 : 00000000 r8 : b4c8c88d
[ 43.536572] r7 : b9362400 r6 : b764f200 r5 : b96b1840 r4 : 00000024
[ 43.537719] r3 : 00000000 r2 : 00000004 r1 : 00000000 r0 : b96b1840
[ 43.538836] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 43.539953] Control: 10c5383d Table: 34da406a DAC: 00000055
[ 43.541092] Process kworker/u8:1 (pid: 44, stack limit = 0xbd776f43)
[ 43.542263] Stack: (0xb9fb3de0 to 0xb9fb4000)
[ 43.543427] 3de0: 80586f94 00000000 b764f200 b764f260 b9fb3e1c b9fb3e00 80570468 80586fa0
[ 43.545736] 3e00: 00000079 00000001 bc349000 bc34b000 b9fb3e84 b9fb3e20 8056e1d8 805703f8
[ 43.548036] 3e20: b764f274 80c92ad8 8014adf4 55555556 ba351e40 80830938 80958488 00000000
[ 43.550327] 3e40: bc34b000 bc349000 00000008 b4c8c895 00000000 00000000 80d07fc0 00000008
[ 43.552690] 3e60: 00000000 b4c8c88d b9362400 b4c8c88d b96a5008 00000000 b9fb3e9c b9fb3e88
[ 43.555180] 3e80: 8056e58c 8056db84 00000001 b9fb3e98 b9fb3eb4 b9fb3ea0 805712bc 8056e574
[ 43.557798] 3ea0: 00000008 00000008 b9fb3ed4 b9fb3eb8 80571dc0 80571298 b4c8c800 b96a5000
[ 43.560523] 3ec0: b96a5014 b96a5004 b9fb3efc b9fb3ed8 80571864 80571d84 b9f49000 b96a5004
[ 43.563383] 3ee0: b9c16400 b9c26000 00000000 b96a5008 b9fb3f34 b9fb3f00 8013bf0c 805717b8
[ 43.566296] 3f00: 40000093 b9c16400 b9c16400 b9c16400 b9f49014 b9c16400 b9c16418 80d03d00
[ 43.569277] 3f20: 00000088 b9f49000 b9fb3f7c b9fb3f38 8013c250 8013bda8 b9fb3f5c 00000000
[ 43.572298] 3f40: 80d03d00 80d03d00 80d8ecfa b9fb2038 b9fb3f7c b9d072c0 b9f39700 00000000
[ 43.575316] 3f60: b9f49000 8013c1f4 b9d072dc b9d35e74 b9fb3fac b9fb3f80 8014253c 8013c200
[ 43.578337] 3f80: 80104378 b9f39700 80142404 00000000 00000000 00000000 00000000 00000000
[ 43.581347] 3fa0: 00000000 b9fb3fb0 801010ac 80142410 00000000 00000000 00000000 00000000
[ 43.584356] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 43.587377] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 43.590406] [<805870ac>] (uart_throttle) from [<80570468>] (tty_throttle_safe+0x7c/0x80)
[ 43.593429] [<80570468>] (tty_throttle_safe) from [<8056e1d8>] (n_tty_receive_buf_common+0x660/0x9f0)
[ 43.596474] [<8056e1d8>] (n_tty_receive_buf_common) from [<8056e58c>] (n_tty_receive_buf2+0x24/0x2c)
[ 43.599511] [<8056e58c>] (n_tty_receive_buf2) from [<805712bc>] (tty_ldisc_receive_buf+0x30/0x6c)
[ 43.602562] [<805712bc>] (tty_ldisc_receive_buf) from [<80571dc0>] (tty_port_default_receive_buf+0x48/0x64)
[ 43.605632] [<80571dc0>] (tty_port_default_receive_buf) from [<80571864>] (flush_to_ldisc+0xb8/0xe8)
[ 43.608711] [<80571864>] (flush_to_ldisc) from [<8013bf0c>] (process_one_work+0x170/0x458)
[ 43.611786] [<8013bf0c>] (process_one_work) from [<8013c250>] (worker_thread+0x5c/0x5a4)
[ 43.614854] [<8013c250>] (worker_thread) from [<8014253c>] (kthread+0x138/0x168)
[ 43.617919] [<8014253c>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[ 43.619495] Exception stack(0xb9fb3fb0 to 0xb9fb3ff8)
[ 43.621042] 3fa0: 00000000 00000000 00000000 00000000
[ 43.624058] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 43.627035] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 43.628548] Code: bad PC value
[ 43.630064] ---[ end trace f3e49c8e83a2d265 ]---

Dominant language
C
Stars
13.2k
Forks
5.5k
Avg merge
2d 21h
Merged PRs (30d)
21

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from raspberrypi/linux

All issues in raspberrypi/linux

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.