V4L2 recurent Exception stack after few hours/days of use, v4l2-ctl freeze, any hardware any Linux version

Start with the reported v4l2-ctl freeze and the kernel log entries for bcm2835_mmal_vchiq, bcm2835-v4l2, and vchiq_release. Reproduce the failure over long-running use on the Raspberry Pi and compare the blocked-task traces. Done means identifying a reproducible cause and confirming that the freeze and timeout messages no longer occur.

Describe the bug
Add a clear and concise description of what you think the bug is.

It's a old hard to get but bug but happens quite often

We use V4L2 on about <= 50 online 24/7 system, any raspbery PI (2/3A+/3B+/4) any camera (v1 any/v2 official/HQ official), Last & clean stable Buster from continuous integration (always up to date but no rpi-update).

cmdline : v4l2-ctl process freeze, never terminate

/var/log/syslog say :

Sep  2 14:06:52 raspberrypi systemd[1683]: Reached target Sockets.
Sep  2 14:06:52 raspberrypi systemd[1683]: Reached target Basic System.
Sep  2 14:06:52 raspberrypi systemd[1683]: Reached target Default.
Sep  2 14:06:52 raspberrypi systemd[1683]: Startup finished in 156ms.
Sep  2 14:06:52 raspberrypi systemd[1]: Started User Manager for UID 0.
Sep  2 14:06:52 raspberrypi systemd[1]: Started Session c7 of user root.
Sep  2 14:07:00 raspberrypi kernel: [184847.809038] bcm2835_mmal_vchiq: timed out waiting for sync completion
Sep  2 14:07:03 raspberrypi kernel: [184850.859089] bcm2835_mmal_vchiq: timed out waiting for sync completion
Sep  2 14:07:19 raspberrypi kernel: [184866.609349] bcm2835_mmal_vchiq: timed out waiting for sync completion
Sep  2 14:07:22 raspberrypi kernel: [184869.649375] bcm2835_mmal_vchiq: timed out waiting for sync completion
Sep  2 14:07:26 raspberrypi kernel: [184873.489445] bcm2835_mmal_vchiq: timed out waiting for sync completion
Sep  2 14:07:29 raspberrypi kernel: [184876.539490] bcm2835_mmal_vchiq: timed out waiting for sync completion
Sep  2 14:07:30 raspberrypi kernel: [184877.567074] bcm2835-v4l2: queue_setup: capture port not configured
Sep  2 14:08:27 raspberrypi kernel: [184935.410453] INFO: task HCEC Notify:31211 blocked for more than 120 seconds.
Sep  2 14:08:27 raspberrypi kernel: [184935.410462]       Tainted: G        WC        4.19.118-v7l+ #1311
Sep  2 14:08:27 raspberrypi kernel: [184935.410468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  2 14:08:27 raspberrypi kernel: [184935.410475] HCEC Notify     D    0 31211  31206 0x00000005
Sep  2 14:08:27 raspberrypi kernel: [184935.410501] [<c09d9ee4>] (__schedule) from [<c09da554>] (schedule+0x50/0xa8)
Sep  2 14:08:27 raspberrypi kernel: [184935.410514] [<c09da554>] (schedule) from [<c09de53c>] (schedule_timeout+0x200/0x428)
Sep  2 14:08:27 raspberrypi kernel: [184935.410524] [<c09de53c>] (schedule_timeout) from [<c09dcad0>] (__down+0x98/0xec)
Sep  2 14:08:27 raspberrypi kernel: [184935.410536] [<c09dcad0>] (__down) from [<c0278e44>] (down+0x54/0x68)
Sep  2 14:08:27 raspberrypi kernel: [184935.410551] [<c0278e44>] (down) from [<c087dfac>] (vchiq_release+0x160/0x354)
Sep  2 14:08:27 raspberrypi kernel: [184935.410566] [<c087dfac>] (vchiq_release) from [<c03d6d04>] (__fput+0xa4/0x1e8)
Sep  2 14:08:27 raspberrypi kernel: [184935.410580] [<c03d6d04>] (__fput) from [<c03d6eb8>] (____fput+0x18/0x1c)
Sep  2 14:08:27 raspberrypi kernel: [184935.410593] [<c03d6eb8>] (____fput) from [<c0242734>] (task_work_run+0xbc/0xe0)
Sep  2 14:08:27 raspberrypi kernel: [184935.410607] [<c0242734>] (task_work_run) from [<c0226bd0>] (do_exit+0x448/0xc14)
Sep  2 14:08:27 raspberrypi kernel: [184935.410619] [<c0226bd0>] (do_exit) from [<c0227434>] (do_group_exit+0x4c/0xe4)
Sep  2 14:08:27 raspberrypi kernel: [184935.410631] [<c0227434>] (do_group_exit) from [<c02337b4>] (get_signal+0x1a4/0x8cc)
Sep  2 14:08:27 raspberrypi kernel: [184935.410643] [<c02337b4>] (get_signal) from [<c020c744>] (do_signal+0x84/0x474)
Sep  2 14:08:27 raspberrypi kernel: [184935.410653] [<c020c744>] (do_signal) from [<c020cd40>] (do_work_pending+0xdc/0xf4)
Sep  2 14:08:27 raspberrypi kernel: [184935.410663] [<c020cd40>] (do_work_pending) from [<c0201034>] (slow_work_pending+0xc/0x20)
Sep  2 14:08:27 raspberrypi kernel: [184935.410670] Exception stack(0xda813fb0 to 0xda813ff8)
Sep  2 14:08:27 raspberrypi kernel: [184935.410678] 3fa0:                                     b6d93384 00000189 00000001 00000000
Sep  2 14:08:27 raspberrypi kernel: [184935.410687] 3fc0: 00000000 ffffffff 00000000 000000f0 be9da3fa b6d926e0 b6d942a8 b51fcf9c
Sep  2 14:08:27 raspberrypi kernel: [184935.410695] 3fe0: 00000000 b51fcdb0 00000000 b6d20088 80000010 b6d93384
Sep  2 14:08:27 raspberrypi kernel: [184935.410704] INFO: task HCEC Notify:31235 blocked for more than 120 seconds.
Sep  2 14:08:27 raspberrypi kernel: [184935.410710]       Tainted: G        WC        4.19.118-v7l+ #1311
Sep  2 14:08:27 raspberrypi kernel: [184935.410715] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  2 14:08:27 raspberrypi kernel: [184935.410721] HCEC Notify     D    0 31235  31230 0x00000005
Sep  2 14:08:27 raspberrypi kernel: [184935.410737] [<c09d9ee4>] (__schedule) from [<c09da554>] (schedule+0x50/0xa8)
Sep  2 14:08:27 raspberrypi kernel: [184935.410748] [<c09da554>] (schedule) from [<c09de53c>] (schedule_timeout+0x200/0x428)
Sep  2 14:08:27 raspberrypi kernel: [184935.410757] [<c09de53c>] (schedule_timeout) from [<c09dcad0>] (__down+0x98/0xec)
Sep  2 14:08:27 raspberrypi kernel: [184935.410766] [<c09dcad0>] (__down) from [<c0278e44>] (down+0x54/0x68)
Sep  2 14:08:27 raspberrypi kernel: [184935.410777] [<c0278e44>] (down) from [<c087dfac>] (vchiq_release+0x160/0x354)
Sep  2 14:08:27 raspberrypi kernel: [184935.410790] [<c087dfac>] (vchiq_release) from [<c03d6d04>] (__fput+0xa4/0x1e8)
Sep  2 14:08:27 raspberrypi kernel: [184935.410801] [<c03d6d04>] (__fput) from [<c03d6eb8>] (____fput+0x18/0x1c)
Sep  2 14:08:27 raspberrypi kernel: [184935.410812] [<c03d6eb8>] (____fput) from [<c0242734>] (task_work_run+0xbc/0xe0)
Sep  2 14:08:27 raspberrypi kernel: [184935.410824] [<c0242734>] (task_work_run) from [<c0226bd0>] (do_exit+0x448/0xc14)
Sep  2 14:08:27 raspberrypi kernel: [184935.410835] [<c0226bd0>] (do_exit) from [<c0227434>] (do_group_exit+0x4c/0xe4)
Sep  2 14:08:27 raspberrypi kernel: [184935.410846] [<c0227434>] (do_group_exit) from [<c02337b4>] (get_signal+0x1a4/0x8cc)
Sep  2 14:08:27 raspberrypi kernel: [184935.410855] [<c02337b4>] (get_signal) from [<c020c744>] (do_signal+0x84/0x474)
Sep  2 14:08:27 raspberrypi kernel: [184935.410864] [<c020c744>] (do_signal) from [<c020cd40>] (do_work_pending+0xdc/0xf4)
Sep  2 14:08:27 raspberrypi kernel: [184935.410874] [<c020cd40>] (do_work_pending) from [<c0201034>] (slow_work_pending+0xc/0x20)
Sep  2 14:08:27 raspberrypi kernel: [184935.410880] Exception stack(0xd40d1fb0 to 0xd40d1ff8)
Sep  2 14:08:27 raspberrypi kernel: [184935.410888] 1fa0:                                     b6dc5384 00000189 00000001 00000000
Sep  2 14:08:27 raspberrypi kernel: [184935.410896] 1fc0: 00000000 ffffffff 00000000 000000f0 bec5e3fa b6dc46e0 b6dc62a8 b51fcf9c
Sep  2 14:08:27 raspberrypi kernel: [184935.410903] 1fe0: 00000000 b51fcdb0 00000000 b6d52088 80000010 b6dc5384
Sep  2 14:08:27 raspberrypi kernel: [184935.410911] INFO: task VCHIQ completio:31526 blocked for more than 120 seconds.
Sep  2 14:08:27 raspberrypi kernel: [184935.410917]       Tainted: G        WC        4.19.118-v7l+ #1311
Sep  2 14:08:27 raspberrypi kernel: [184935.410923] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  2 14:08:27 raspberrypi kernel: [184935.410928] VCHIQ completio D    0 31526  31524 0x00000005
Sep  2 14:08:27 raspberrypi kernel: [184935.410944] [<c09d9ee4>] (__schedule) from [<c09da554>] (schedule+0x50/0xa8)
Sep  2 14:08:27 raspberrypi kernel: [184935.410954] [<c09da554>] (schedule) from [<c09de53c>] (schedule_timeout+0x200/0x428)
Sep  2 14:08:27 raspberrypi kernel: [184935.410963] [<c09de53c>] (schedule_timeout) from [<c09dcad0>] (__down+0x98/0xec)
Sep  2 14:08:27 raspberrypi kernel: [184935.410972] [<c09dcad0>] (__down) from [<c0278e44>] (down+0x54/0x68)
Sep  2 14:08:27 raspberrypi kernel: [184935.410983] [<c0278e44>] (down) from [<c087dfac>] (vchiq_release+0x160/0x354)
Sep  2 14:08:27 raspberrypi kernel: [184935.410995] [<c087dfac>] (vchiq_release) from [<c03d6d04>] (__fput+0xa4/0x1e8)
Sep  2 14:08:27 raspberrypi kernel: [184935.411006] [<c03d6d04>] (__fput) from [<c03d6eb8>] (____fput+0x18/0x1c)
Sep  2 14:08:27 raspberrypi kernel: [184935.411017] [<c03d6eb8>] (____fput) from [<c0242734>] (task_work_run+0xbc/0xe0)
Sep  2 14:08:27 raspberrypi kernel: [184935.411029] [<c0242734>] (task_work_run) from [<c0226bd0>] (do_exit+0x448/0xc14)
Sep  2 14:08:27 raspberrypi kernel: [184935.411040] [<c0226bd0>] (do_exit) from [<c0227434>] (do_group_exit+0x4c/0xe4)
Sep  2 14:08:27 raspberrypi kernel: [184935.411051] [<c0227434>] (do_group_exit) from [<c02337b4>] (get_signal+0x1a4/0x8cc)
Sep  2 14:08:27 raspberrypi kernel: [184935.411060] [<c02337b4>] (get_signal) from [<c020c934>] (do_signal+0x274/0x474)
Sep  2 14:08:27 raspberrypi kernel: [184935.411069] [<c020c934>] (do_signal) from [<c020cd40>] (do_work_pending+0xdc/0xf4)
Sep  2 14:08:27 raspberrypi kernel: [184935.411078] [<c020cd40>] (do_work_pending) from [<c0201034>] (slow_work_pending+0xc/0x20)
Sep  2 14:08:27 raspberrypi kernel: [184935.411085] Exception stack(0xda1a3fb0 to 0xda1a3ff8)
Sep  2 14:08:27 raspberrypi kernel: [184935.411092] 3fa0:                                     fffffffc c014c407 b6b7fd74 00000008
Sep  2 14:08:27 raspberrypi kernel: [184935.411100] 3fc0: b62076e0 b6d402ec b6b814d0 00000036 b6d40308 b6b7fd74 b6d402b4 b6b7ff9c
Sep  2 14:08:27 raspberrypi kernel: [184935.411108] 3fe0: b6d40240 b6b7fd44 b6d2d010 b6c5251c 20000010 00000003
Sep  2 14:08:27 raspberrypi kernel: [184935.411115] INFO: task VCHIQ completio:31654 blocked for more than 120 seconds.
Sep  2 14:08:27 raspberrypi kernel: [184935.411122]       Tainted: G        WC        4.19.118-v7l+ #1311
Sep  2 14:08:27 raspberrypi kernel: [184935.411127] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  2 14:08:27 raspberrypi kernel: [184935.411133] VCHIQ completio D    0 31654  31652 0x00000005
Sep  2 14:08:27 raspberrypi kernel: [184935.411148] [<c09d9ee4>] (__schedule) from [<c09da554>] (schedule+0x50/0xa8)
Sep  2 14:08:27 raspberrypi kernel: [184935.411158] [<c09da554>] (schedule) from [<c09de53c>] (schedule_timeout+0x200/0x428)
Sep  2 14:08:27 raspberrypi kernel: [184935.411167] [<c09de53c>] (schedule_timeout) from [<c09dcad0>] (__down+0x98/0xec)
Sep  2 14:08:27 raspberrypi kernel: [184935.411176] [<c09dcad0>] (__down) from [<c0278e44>] (down+0x54/0x68)
Sep  2 14:08:27 raspberrypi kernel: [184935.411187] [<c0278e44>] (down) from [<c087dfac>] (vchiq_release+0x160/0x354)
Sep  2 14:08:27 raspberrypi kernel: [184935.411199] [<c087dfac>] (vchiq_release) from [<c03d6d04>] (__fput+0xa4/0x1e8)
Sep  2 14:08:27 raspberrypi kernel: [184935.411210] [<c03d6d04>] (__fput) from [<c03d6eb8>] (____fput+0x18/0x1c)
Sep  2 14:08:27 raspberrypi kernel: [184935.411221] [<c03d6eb8>] (____fput) from [<c0242734>] (task_work_run+0xbc/0xe0)
Sep  2 14:08:27 raspberrypi kernel: [184935.411233] [<c0242734>] (task_work_run) from [<c0226bd0>] (do_exit+0x448/0xc14)
Sep  2 14:08:27 raspberrypi kernel: [184935.411244] [<c0226bd0>] (do_exit) from [<c0227434>] (do_group_exit+0x4c/0xe4)
Sep  2 14:08:27 raspberrypi kernel: [184935.411255] [<c0227434>] (do_group_exit) from [<c02337b4>] (get_signal+0x1a4/0x8cc)
Sep  2 14:08:27 raspberrypi kernel: [184935.411265] [<c02337b4>] (get_signal) from [<c020c934>] (do_signal+0x274/0x474)
Sep  2 14:08:27 raspberrypi kernel: [184935.411273] [<c020c934>] (do_signal) from [<c020cd40>] (do_work_pending+0xdc/0xf4)
Sep  2 14:08:27 raspberrypi kernel: [184935.411282] [<c020cd40>] (do_work_pending) from [<c0201034>] (slow_work_pending+0xc/0x20)
Sep  2 14:08:27 raspberrypi kernel: [184935.411289] Exception stack(0xda1abfb0 to 0xda1abff8)
Sep  2 14:08:27 raspberrypi kernel: [184935.411296] bfa0:                                     fffffffc c014c407 b6c59d74 00000008
Sep  2 14:08:27 raspberrypi kernel: [184935.411304] bfc0: b63076e0 b6e1a2ec b6c5b4d0 00000036 b6e1a308 b6c59d74 b6e1a2b4 b6c59f9c
Sep  2 14:08:27 raspberrypi kernel: [184935.411312] bfe0: b6e1a240 b6c59d44 b6e07010 b6d2c51c 20000010 00000003
Sep  2 14:08:27 raspberrypi kernel: [184935.411320] INFO: task VCHIQ completio:31691 blocked for more than 120 seconds.
Sep  2 14:08:27 raspberrypi kernel: [184935.411326]       Tainted: G        WC        4.19.118-v7l+ #1311
Sep  2 14:08:27 raspberrypi kernel: [184935.411331] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Stock Buster Raspbian

To reproduce
List the steps required to reproduce the issue.

Multiple automatic call of v4l2-ctl / use of V4L2 every days.

Expected behaviour
Add a clear and concise description of what you expected to happen.

No crash

Actual behaviour
Add a clear and concise description of what actually happened.

v4l2-ctl freeze and syslog Exception stack

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW

All Raspbery PI from 2 to 4 (RPI 2 / 3A+ / 3B+ / 4)

• Which OS and version (cat /etc/rpi-issue)?

Last Buster, same bug since Jessie.

• Which firmware version (vcgencmd version)?

All since few years.

• Which kernel version (uname -a)?

Last / All since Jessie

Logs
If applicable, add the relevant output from dmesg or similar.

Complete /var/log/syslog here :
https://www.serveurperso.com/temp/syslog

Additional context
Add any other relevant context for the problem.

I'm Available for any help and debug. I keep a buggy system without reboot for help.