soft device error: 0004 when paring with BLE HID peripheral

I'm trying to get the BLE HID central to play (I just need the event report bytes). HID needs pairing and when I try to pair I get the following error and then the BT connection just times out:

```
Traceback (most recent call last):
File "code.py", line 40, in
File "adafruit_ble/__init__.py", line 143, in pair
_bleio.BluetoothError: Unknown soft device error: 0004
```

The code I use is the following
```python
import _bleio
import board
import time
# import usb_hid

from adafruit_ble import BLERadio
from adafruit_ble.advertising.standard import ProvideServicesAdvertisement
from adafruit_ble.services.standard.hid import HIDService

# keyboard = usb_hid.devices[0]

# keyboard.send_report()

ble = BLERadio()
bleHid = None

# _bleio.adapter.erase_bonding()
if ble.connected:
for conn in ble.connections:
conn.disconnect()
# if HIDService in conn:
# bleHid = conn
# break

while True:
if not bleHid:
print("Scanning...")
for adv in ble.start_scan(ProvideServicesAdvertisement, timeout=5, extended=True):
print(f"Discovered '{adv.complete_name}'")

conn = ble.connect(adv)
if HIDService in conn:
print("Found HID service advertisement!")
else:
conn.disconnect()
continue

if not conn.paired:
print("Pairing...")
conn.pair()
print("Paired")

print("Putting keyboard in boot report mode")
print(conn[HIDService].protocol_mode)
conn[HIDService].protocol_mode = 0
print(conn[HIDService].protocol_mode)

bleHid = conn

break

ble.stop_scan()

while bleHid and bleHid.connected:
# print(bleHid[HIDService].boot_keyboard_out)
data = bleHid[HIDService].boot_keyboard_out
# data = bleHid[HIDService].devices[0].report
# print(conn._constructed_services)
data_int = int.from_bytes(data, "big")
# print(data_int)
if data_int != 0:
print(data)
# time.sleep(0.2)

time.sleep(0.2)
```

I tried `# _bleio.adapter.erase_bonding()` just in case the flash is full at the address that pairings take place but it didn't work (same exact error). I'm starting to think that this is a circuitpython error?

I'm on an nrf52840 btw and the peripheral is known to work flawlessly.