Cannot pair to Google Home speakers

This is probably bigger than these specific devices, but I noticed it specifically on a couple of Google Home speakers (specifically Nest Minis and Google Home Minis).

The speakers have a Bluetooth pairing mode. When enabling it, the speaker shows up in the plug, and pairing appears to work at first, but then I always get "Unable to Connect" afterwards. However, using blueman I can actually pair and connect.

Blueman showed something about "connecting to the audio sync" when pairing, which might be related?