Hostapd+brcmfmac stuck kernel on load on wlan0 interface

Start with the brcmfmac SDIO and MMC paths named in the trace, including brcmf_sdio_watchdog_thread, brcmf_sdio_dataworker, and mmc_wait_for_req. Reproduce on the stated Raspberry Pi 3B setup and kernel 4.14.79, then identify a reproducible root cause and a verified kernel or configuration fix for the hang.

I encountered an random stuck of the kernel on Avnet device (https://fr.farnell.com/avnet/avtse-rpi-iiotg/kit-gateway-iot-arm-cortex-a53/dp/3105658) based on raspberry pi3b board.
I'm using hostapd to provide AP mode to my customers .
The clients can connect successfully to the Wifi and recover they ip address (The AP does not provide internet access by only an web application running on my raspbian ) .
Sometime , when accessing the web page the Wifi disconnect and the kernel is stuck with only this message in dmesg :
mmc1: Timeout waiting for hardware interrupt.

Few seconds later i had this error

[10074.191524]       Not tainted 4.14.79-fnd-201214-v7+ #8
[10074.191532] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[10074.191540] brcmf_wdog/mmc1 D    0   253      2 0x00000000
[10074.191591] [<808172b4>] (__schedule) from [<80817a18>] (schedule+0x50/0xa8)
[10074.191612] [<80817a18>] (schedule) from [<8068c270>] (__mmc_claim_host+0xa4/0x1dc)
[10074.191638] [<8068c270>] (__mmc_claim_host) from [<80699818>] (sdio_claim_host+0x2c/0x44)
[10074.191855] [<80699818>] (sdio_claim_host) from [<7f2ee2f8>] (brcmf_sdio_watchdog_thread+0x198/0x298 [brcmfmac])
[10074.192045] [<7f2ee2f8>] (brcmf_sdio_watchdog_thread [brcmfmac]) from [<80140074>] (kthread+0x144/0x174)
[10074.192066] [<80140074>] (kthread) from [<8010806c>] (ret_from_fork+0x14/0x28)
[10074.192219] INFO: task kworker/u8:2:2681 blocked for more than 120 seconds.
[10074.192226]       Not tainted 4.14.79-fnd-201214-v7+ #8
[10074.192230] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[10074.192235] kworker/u8:2    D    0  2681      2 0x00000000
[10074.192402] Workqueue: brcmf_wq/mmc1:0001:1 brcmf_sdio_dataworker [brcmfmac]
[10074.192449] [<808172b4>] (__schedule) from [<80817a18>] (schedule+0x50/0xa8)
[10074.192465] [<80817a18>] (schedule) from [<8081b570>] (schedule_timeout+0x2e4/0x410)
[10074.192480] [<8081b570>] (schedule_timeout) from [<80818678>] (wait_for_common+0xbc/0x170)
[10074.192497] [<80818678>] (wait_for_common) from [<8081874c>] (wait_for_completion+0x20/0x24)
[10074.192516] [<8081874c>] (wait_for_completion) from [<8068c0fc>] (mmc_wait_for_req_done+0x9c/0x118)
[10074.192534] [<8068c0fc>] (mmc_wait_for_req_done) from [<8068d068>] (mmc_wait_for_req+0xd0/0xfc)
[10074.192553] [<8068d068>] (mmc_wait_for_req) from [<806987a4>] (mmc_io_rw_extended+0x22c/0x2c8)
[10074.192571] [<806987a4>] (mmc_io_rw_extended) from [<80699f24>] (sdio_io_rw_ext_helper+0x11c/0x220)
[10074.192584] [<80699f24>] (sdio_io_rw_ext_helper) from [<8069a158>] (sdio_memcpy_toio+0x38/0x40)
[10074.192744] [<8069a158>] (sdio_memcpy_toio) from [<7f2ef624>] (brcmf_sdiod_buffrw+0x9c/0xa4 [brcmfmac])
[10074.193053] [<7f2ef624>] (brcmf_sdiod_buffrw [brcmfmac]) from [<7f2f08d4>] (brcmf_sdiod_send_pkt+0xac/0xec [brcmfmac])
[10074.193361] [<7f2f08d4>] (brcmf_sdiod_send_pkt [brcmfmac]) from [<7f2ed64c>] (brcmf_sdio_dataworker+0x17ac/0x22c0 [brcmfmac])
[10074.193521] [<7f2ed64c>] (brcmf_sdio_dataworker [brcmfmac]) from [<801398cc>] (process_one_work+0x1e8/0x4dc)
[10074.193539] [<801398cc>] (process_one_work) from [<8013a8c4>] (worker_thread+0x54/0x57c)
[10074.193553] [<8013a8c4>] (worker_thread) from [<80140074>] (kthread+0x144/0x174)
[10074.193571] [<80140074>] (kthread) from [<8010806c>] (ret_from_fork+0x14/0x28)
[10166.352562] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b3ee782e len 74
[10166.352582] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 11
[10196.303940] brcmfmac: brcmf_cfg80211_get_station Enter, MAC f6:54:e0:cc:d5:fb
[10196.303960] brcmfmac: brcmf_sdio_bus_txctl Enter
[10197.073059] INFO: task brcmf_wdog/mmc1:253 blocked for more than 120 seconds.

The only way to go away is hard reboot.
I tried to update brcm80211 to the latest firmware whith the same behaviour .
Does someone has an idea what could be the root cause of this issue.

Kernel : 4.14.79
RASPBIAN VERSION="9 (stretch)"
Hostapd = 2:2.6-21bpo9rpt1
brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: Oct 23 2017 03:55:53 version 7.45.98.38 (r674442 CY) FWID 01-e58d219f
brcmfmac: brcmf_c_preinit_dcmds: CLM version = API: 12.2 Data: 7.11.15 Compiler: 1.24.2 ClmImport: 1.24.1 Creation: 2014-05-26 10:53:55 Inc Data: 9.10.39 Inc Compiler: 1.29.4 Inc ClmImport: 1.36.3 Creation: 2017-10-23 03:47:14

I set the module in debug mode, you will find below the traces

cat /sys/kernel/debug/tracing/trace

# tracer: nop
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
         hostapd-666   [002] ....  7227.266892: rdev_add_key: phy0, netdev:wlan0(5), key_index: 1, pairwise: false, mac addr: 00:00:00:00:00:00
         hostapd-666   [002] ....  7227.271835: rdev_return_int: phy0, returned: 0
         hostapd-666   [002] ....  7227.271947: rdev_set_default_key: phy0, netdev:wlan0(5), key index: 1, unicast: false, multicast: true
         hostapd-666   [002] ....  7227.272517: rdev_return_int: phy0, returned: 0
         hostapd-666   [002] ....  7473.579939: rdev_del_station: phy0, netdev:wlan0(5), station mac: ff:ff:ff:ff:ff:ff, subtype: 12, reason_code: 3
         hostapd-666   [002] ....  7473.585972: rdev_return_int: phy0, returned: 0
         hostapd-666   [002] ....  7473.640557: rdev_stop_ap: phy0, netdev:wlan0(5)
         hostapd-666   [002] ....  7474.101668: rdev_return_int: phy0, returned: 0
         hostapd-666   [002] ....  7474.140546: rdev_change_virtual_intf: phy0, netdev:wlan0(5), type: 2
         hostapd-666   [002] ....  7474.144124: rdev_return_int: phy0, returned: 0
        iwconfig-26650 [003] ....  7479.039279: rdev_get_tx_power: phy0, wdev(1)
        iwconfig-26650 [003] ....  7479.039289: rdev_return_int_int: phy0, function returns: -5, function filled: 2132311892
        iwconfig-26690 [002] ....  7489.153372: rdev_get_tx_power: phy0, wdev(1)
        iwconfig-26690 [002] ....  7489.153384: rdev_return_int_int: phy0, function returns: -5, function filled: 2132311892
              ip-26697 [000] ....  7489.198729: rdev_set_power_mgmt: phy0, netdev:wlan0(5), not enabled, timeout: -1 
              ip-26697 [000] .n..  7489.199982: rdev_return_int: phy0, returned: 0
         hostapd-2620  [001] ....  9877.762736: rdev_get_channel: phy0, wdev(1)
         hostapd-2620  [001] ....  9877.766625: rdev_return_chandef: phy0, band: 0, control freq: 2437, width: 1, cf1: 2437, cf2: 0, ret: 0
         hostapd-2620  [001] ....  9877.766634: rdev_get_tx_power: phy0, wdev(1)
         hostapd-2620  [001] ....  9877.767108: rdev_return_int_int: phy0, function returns: 0, function filled: 31
         hostapd-2620  [001] ....  9877.767228: rdev_get_channel: phy0, wdev(1)
         hostapd-2620  [001] ....  9877.767899: rdev_return_chandef: phy0, band: 0, control freq: 2437, width: 1, cf1: 2437, cf2: 0, ret: 0
         hostapd-2620  [001] ....  9877.767903: rdev_get_tx_power: phy0, wdev(1)
         hostapd-2620  [001] ....  9877.768348: rdev_return_int_int: phy0, function returns: 0, function filled: 31
         hostapd-2620  [001] ....  9877.768930: rdev_get_channel: phy0, wdev(1)
         hostapd-2620  [003] ....  9877.779104: rdev_return_chandef: phy0, band: 0, control freq: 2437, width: 1, cf1: 2437, cf2: 0, ret: 0
         hostapd-2620  [003] ....  9877.779123: rdev_get_tx_power: phy0, wdev(1)
         hostapd-2620  [003] ....  9877.780288: rdev_return_int_int: phy0, function returns: 0, function filled: 31
         hostapd-2620  [003] ....  9877.780481: rdev_change_virtual_intf: phy0, netdev:wlan0(5), type: 3
         hostapd-2620  [003] ....  9877.780511: rdev_return_int: phy0, returned: 0
         hostapd-2620  [003] ....  9877.781742: rdev_get_channel: phy0, wdev(1)
         hostapd-2620  [003] ....  9877.782545: rdev_return_chandef: phy0, band: 0, control freq: 2437, width: 1, cf1: 2437, cf2: 0, ret: 0
         hostapd-2620  [003] ....  9877.782555: rdev_get_tx_power: phy0, wdev(1)
         hostapd-2620  [003] ....  9877.783032: rdev_return_int_int: phy0, function returns: 0, function filled: 31
         hostapd-2620  [003] ....  9877.785382: cfg80211_reg_can_beacon: phy0, band: 0, control freq: 2437, width: 1, cf1: 2437, cf2: 0, iftype=3 check_no_ir=true
         hostapd-2620  [003] ....  9877.785393: cfg80211_return_bool: returned true
         hostapd-2620  [003] ....  9877.785481: rdev_del_station: phy0, netdev:wlan0(5), station mac: 00:00:00:00:00:00, subtype: 12, reason_code: 2
         hostapd-2620  [003] ....  9877.785486: rdev_return_int: phy0, returned: -14
         hostapd-2620  [003] ....  9877.785597: rdev_del_station: phy0, netdev:wlan0(5), station mac: ff:ff:ff:ff:ff:ff, subtype: 12, reason_code: 2
         hostapd-2620  [003] ....  9877.787043: rdev_return_int: phy0, returned: 0
         hostapd-2620  [003] ....  9877.787281: rdev_del_key: phy0, netdev:wlan0(5), key_index: 0, pairwise: false, mac addr: 00:00:00:00:00:00
         hostapd-2620  [003] ....  9877.787296: rdev_return_int: phy0, returned: -22
         hostapd-2620  [003] ....  9877.787390: rdev_del_key: phy0, netdev:wlan0(5), key_index: 1, pairwise: false, mac addr: 00:00:00:00:00:00
         hostapd-2620  [003] ....  9877.792142: rdev_return_int: phy0, returned: 0
         hostapd-2620  [003] ....  9877.792438: rdev_del_key: phy0, netdev:wlan0(5), key_index: 2, pairwise: false, mac addr: 00:00:00:00:00:00
         hostapd-2620  [003] ....  9877.793094: rdev_return_int: phy0, returned: 0
         hostapd-2620  [003] ....  9877.793284: rdev_del_key: phy0, netdev:wlan0(5), key_index: 3, pairwise: false, mac addr: 00:00:00:00:00:00
         hostapd-2620  [003] ....  9877.793296: rdev_return_int: phy0, returned: -22
         hostapd-2620  [003] ....  9877.879009: cfg80211_reg_can_beacon: phy0, band: 0, control freq: 2437, width: 1, cf1: 2437, cf2: 0, iftype=3 check_no_ir=true
         hostapd-2620  [003] ....  9877.879019: cfg80211_return_bool: returned true
         hostapd-2620  [003] ....  9877.879024: rdev_start_ap: phy0, netdev:wlan0(5), AP settings - ssid: SSID-147, band: 0, control freq: 2437, width: 1, cf1: 2437, cf2: 0, beacon interval: 100, dtim period: 2, hidden ssid: 0, wpa versions: 2, privacy: true, auth type: 0, inactivity timeout: 0
         hostapd-2620  [003] .n..  9878.016361: rdev_return_int: phy0, returned: 0
         hostapd-2620  [003] ....  9878.017502: rdev_add_key: phy0, netdev:wlan0(5), key_index: 1, pairwise: false, mac addr: 00:00:00:00:00:00
         hostapd-2620  [003] ....  9878.031091: rdev_return_int: phy0, returned: 0
         hostapd-2620  [003] ....  9878.031225: rdev_set_default_key: phy0, netdev:wlan0(5), key index: 1, unicast: false, multicast: true
         hostapd-2620  [003] ....  9878.031798: rdev_return_int: phy0, returned: 0
     kworker/2:1-32161 [002] ....  9896.283476: cfg80211_new_sta: netdev:wlan0(5), f6:54:e0:cc:d5:fb
         hostapd-2623  [000] ....  9896.285387: rdev_change_station: phy0, netdev:wlan0(5), station mac: f6:54:e0:cc:d5:fb, station flags mask: 28, station flags set: 0, station modify mask: 0, listen interval: -1, aid: 0, plink action: 0, plink state: 0, uapsd queues: 0, vlan:
         hostapd-2623  [000] ....  9896.285441: rdev_return_int: phy0, returned: 0
         hostapd-2623  [000] ....  9896.285712: rdev_del_key: phy0, netdev:wlan0(5), key_index: 0, pairwise: true, mac addr: f6:54:e0:cc:d5:fb
         hostapd-2623  [000] ....  9896.285727: rdev_return_int: phy0, returned: -22
         hostapd-2623  [000] ....  9896.285914: rdev_del_key: phy0, netdev:wlan0(5), key_index: 0, pairwise: true, mac addr: f6:54:e0:cc:d5:fb
         hostapd-2623  [000] ....  9896.285941: rdev_return_int: phy0, returned: -22
         hostapd-2623  [000] ....  9896.286006: rdev_change_station: phy0, netdev:wlan0(5), station mac: f6:54:e0:cc:d5:fb, station flags mask: 2, station flags set: 0, station modify mask: 0, listen interval: -1, aid: 0, plink action: 0, plink state: 0, uapsd queues: 0, vlan:
         hostapd-2623  [002] ....  9896.286754: rdev_return_int: phy0, returned: 0
         hostapd-2623  [002] ....  9896.288354: rdev_add_key: phy0, netdev:wlan0(5), key_index: 1, pairwise: false, mac addr: 00:00:00:00:00:00
         hostapd-2623  [002] ....  9896.290349: rdev_return_int: phy0, returned: 0
         hostapd-2623  [002] ....  9896.290496: rdev_set_default_key: phy0, netdev:wlan0(5), key index: 1, unicast: false, multicast: true
         hostapd-2623  [002] ....  9896.291155: rdev_return_int: phy0, returned: 0
         hostapd-2623  [002] ....  9896.317917: rdev_get_key: phy0, netdev:wlan0(5), key_index: 1, pairwise: false, mac addr: 00:00:00:00:00:00
         hostapd-2623  [002] ....  9896.318628: rdev_return_int: phy0, returned: 0
         hostapd-2623  [002] ....  9896.335467: rdev_add_key: phy0, netdev:wlan0(5), key_index: 0, pairwise: true, mac addr: f6:54:e0:cc:d5:fb
         hostapd-2623  [002] ....  9896.336151: rdev_return_int: phy0, returned: 0
         hostapd-2623  [002] ....  9896.336292: rdev_change_station: phy0, netdev:wlan0(5), station mac: f6:54:e0:cc:d5:fb, station flags mask: 2, station flags set: 2, station modify mask: 0, listen interval: -1, aid: 0, plink action: 0, plink state: 0, uapsd queues: 0, vlan:
         hostapd-2623  [002] ....  9896.336707: rdev_return_int: phy0, returned: 0

brcmfmac debug output

[ 9943.133193] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data 8aa510ea len 1518
[ 9943.133200] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 0
[ 9943.133230] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data 8aa570ea len 1518
[ 9943.133237] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 1
[ 9943.133256] brcmfmac: brcmf_sdio_dpc Enter
[ 9943.138620] brcmfmac: brcmf_sdio_isr Enter
[ 9943.138800] brcmfmac: brcmf_sdio_dpc Enter
[ 9943.138813] brcmfmac: brcmf_sdio_readframes Enter
[ 9943.139219] brcmfmac: brcmf_sdio_sendfromq Enter
[ 9943.139229] brcmfmac: brcmf_sdio_txpkt Enter
[ 9943.139613] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data 8aa550ea len 1518
[ 9943.139622] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 0
[ 9943.139649] brcmfmac: brcmf_sdio_dpc Enter
[ 9943.139692] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data 8aa538ea len 1518
[ 9943.139698] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 1
[ 9943.139721] brcmfmac: brcmf_sdio_dpc Enter
[ 9943.157015] brcmfmac: brcmf_sdio_isr Enter
[ 9943.157169] brcmfmac: brcmf_sdio_dpc Enter
[ 9943.157185] brcmfmac: brcmf_sdio_readframes Enter
[ 9943.158464] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b060d0ea len 1518
[ 9943.158472] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 2
[ 9943.158580] brcmfmac: brcmf_sdio_sendfromq Enter
[ 9943.158589] brcmfmac: brcmf_sdio_txpkt Enter
[ 9944.589863] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b07830ea len 1518
[ 9944.589882] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 2
[ 9944.829825] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b06088ea len 1518
[ 9944.829832] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 3
[ 9946.189864] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b3ee6c2e len 74
[ 9946.189876] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 4
[ 9953.229902] mmc1: Timeout waiting for hardware interrupt.
[10001.374453] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b01000ea len 101
[10001.374463] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 5
[10001.374713] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b5c2d8ea len 70
[10001.374721] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 6
[10006.430541] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b5c2cc2e len 46
[10006.430550] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 7
[10007.470568] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b5c2da2e len 46
[10007.470608] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 8
[10008.510571] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b5c2d42e len 46
[10008.510617] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 9
[10022.990778] brcmfmac: brcmf_sdio_bus_txdata Enter: pkt: data b3ee7e2e len 74
[10022.990830] brcmfmac: brcmf_sdio_bus_txdata deferring pktq len 10