WiFi component causes boot loop on inkplate6 / ESP32-D0WDQ6 (dual core)

### The problem

I can reliably generate a boot loop with the WiFi component on the inkplate6. This occurs on both arduino and esp-idf frameworks. Removing the wifi component allows the device to boot.

I tried the solution to https://github.com/esphome/issues/issues/4830 but that didn't make a difference.

### Which version of ESPHome has the issue?

2023.8.0

### What type of installation are you using?

Docker

### Which version of Home Assistant has the issue?

_No response_

### What platform are you using?

ESP8266

### Board

inkplate6

### Component causing the issue

wifi

### Example YAML snippet

```yaml
esphome:
name: inkplate6plus
friendly_name: inkplate6plus

esp32:
board: esp-wrover-kit
framework:
type: esp-idf
advanced:
ignore_efuse_mac_crc: true

wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
domain: .b20.iglauer.org

# Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "${friendly_name} AP"
password: !secret fallback_ap_password

#packages:
# device_base: !include aa_common/boilerplate.yaml

logger:
level: DEBUG
```

### Anything in the logs that might be useful for us?

```txt
Arduino:
[I][logger:262]: Log initialized
[I][app:029]: Running through setup()...
[C][wifi:038]: Setting up WiFi...
[C][wifi:048]: Starting WiFi...
[C][wifi:049]: Local MAC: 3C:71:BF:05:F4:B4
ets Jun 8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x32 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:13132
load:0x40080400,len:3036
entry 0x400805e4

ESP-IDF:

I (28) boot: ESP-IDF 4.4.5 2nd stage bootloader
I (29) boot: compile time 11:53:35
I (29) boot: chip revision: v1.0
I (32) boot.esp32: SPI Speed : 40MHz
I (36) boot.esp32: SPI Mode : DIO
I (41) boot.esp32: SPI Flash Size : 4MB
I (45) boot: Enabling RNG early entropy source...
I (51) boot: Partition Table:
I (54) boot: ## Label Usage Type ST Offset Length
I (62) boot: 0 otadata OTA data 01 00 00009000 00002000
I (69) boot: 1 phy_init RF data 01 01 0000b000 00001000
I (76) boot: 2 app0 OTA app 00 10 00010000 001c0000
I (84) boot: 3 app1 OTA app 00 11 001d0000 001c0000
I (91) boot: 4 nvs WiFi data 01 02 00390000 0006d000
I (99) boot: End of partition table
I (103) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=16898h ( 92312) map
I (145) esp_image: segment 1: paddr=000268c0 vaddr=3ffb0000 size=02ed4h ( 11988) load
I (150) esp_image: segment 2: paddr=0002979c vaddr=40080000 size=0687ch ( 26748) load
I (162) esp_image: segment 3: paddr=00030020 vaddr=400d0020 size=70ce4h (462052) map
I (329) esp_image: segment 4: paddr=000a0d0c vaddr=4008687c size=0d798h ( 55192) load
I (362) boot: Loaded app from partition at offset 0x10000
I (362) boot: Disabling RNG early entropy source...
I (374) cpu_start: Pro cpu up.
I (374) cpu_start: Starting app cpu, entry point is 0x4008185c
I (0) cpu_start: App cpu up.
I (388) cpu_start: Pro cpu start user code
I (388) cpu_start: cpu freq: 160000000
I (388) cpu_start: Application information:
I (392) cpu_start: Project name: inkplate6plus
I (398) cpu_start: App version: 2023.8.0
I (403) cpu_start: Compile time: Sep 2 2023 11:52:38
I (409) cpu_start: ELF file SHA256: 855b3860a0b73199...
I (415) cpu_start: ESP-IDF: 4.4.5
I (420) cpu_start: Min chip rev: v0.0
I (424) cpu_start: Max chip rev: v3.99
I (429) cpu_start: Chip rev: v1.0
I (434) heap_init: Initializing. RAM available for dynamic allocation:
I (441) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (447) heap_init: At 3FFB6DB8 len 00029248 (164 KiB): DRAM
I (454) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (460) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (466) heap_init: At 40094014 len 0000BFEC (47 KiB): IRAM
I (474) spi_flash: detected chip: gd
I (477) spi_flash: flash io: dio
I (482) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
[I][logger:262]: Log initialized
[I][app:029]: Running through setup()...
[C][wifi:038]: Setting up WiFi...
[D][esp-idf:000]: I (210) wifi:
[D][esp-idf:000]: wifi driver task: 3ffc3660, prio:23, stack:6656, core=0
[D][esp-idf:000]:

[D][esp-idf:000]: I (918) wifi:
[D][esp-idf:000]: wifi firmware version: 0f80fa0
[D][esp-idf:000]:

[D][esp-idf:000]: I (928) wifi:
[D][esp-idf:000]: wifi certification version: v7.0
[D][esp-idf:000]:

[D][esp-idf:000]: I (938) wifi:
[D][esp-idf:000]: config NVS flash: enabled
[D][esp-idf:000]:

[D][esp-idf:000]: I (949) wifi:
[D][esp-idf:000]: config nano formating: disabled
[D][esp-idf:000]:

[D][esp-idf:000]: I (959) wifi:
[D][esp-idf:000]: Init data frame dynamic rx buffer num: 32
[D][esp-idf:000]:

[D][esp-idf:000]: I (969) wifi:
[D][esp-idf:000]: Init management frame dynamic rx buffer num: 32
[D][esp-idf:000]:

[D][esp-idf:000]: I (979) wifi:
[D][esp-idf:000]: Init management short buffer num: 32
[D][esp-idf:000]:

[D][esp-idf:000]: I (1000) wifi:
[D][esp-idf:000]: Init dynamic tx buffer num: 32
[D][esp-idf:000]:

[D][esp-idf:000]: I (1010) wifi:
[D][esp-idf:000]: Init static rx buffer size: 1600
[D][esp-idf:000]:

[D][esp-idf:000]: I (1020) wifi:
[D][esp-idf:000]: Init static rx buffer num: 10
[D][esp-idf:000]:

[D][esp-idf:000]: I (1031) wifi:
[D][esp-idf:000]: Init dynamic rx buffer num: 32
[D][esp-idf:000]:

[D][esp-idf:000]: I (1042) wifi_init: rx ba win: 6

[D][esp-idf:000]: I (1042) wifi_init: tcpip mbox: 32

[D][esp-idf:000]: I (1051) wifi_init: udp mbox: 6

[D][esp-idf:000]: I (1061) wifi_init: tcp mbox: 6

[D][esp-idf:000]: I (1062) wifi_init: tcp tx win: 5744

[D][esp-idf:000]: I (1072) wifi_init: tcp rx win: 5744

[D][esp-idf:000]: I (1082) wifi_init: tcp mss: 1440

[D][esp-idf:000]: I (1082) wifi_init: WiFi IRAM OP enabled

[D][esp-idf:000]: I (1092) wifi_init: WiFi RX IRAM OP enabled

[C][wifi:048]: Starting WiFi...
[C][wifi:049]: Local MAC: 3C:71:BF:05:F4:B4
[D][esp-idf:000]: I (1114) phy_init: phy_version 4670,719f9f6,Feb 18 2021,17:07:07

ets Jun 8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x32 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:6608
load:0x40078000,len:15060
ho 0 tail 12 room 4
load:0x40080400,len:3816
entry 0x40080698
```

### Additional information

_No response_