Gets stuck breaking blocks on 2b

## Some information
Operating system: Windows 10
Java version: Eclipse Adoptium 17.0.6.10
Minecraft version: 1.20.1
Baritone version: 1.10.1
Other mods (if used):

## Exception, error or logs
Sometimes gets stuck breaking the same block repeatedly.
- Happens with Efficiency 5 netherite pickaxe
- Does not happen with stone pickaxe
- Does not happen with diamond pickaxe

## How to reproduce
On 2b2t:
Obtain efficiency 5 netherite pickaxe
Set positions 1 and 2.
Clear the area (set air)
```
#sel p1
#sel p2
#sel set air
```
## Modified settings
None

## Final checklist
- [x] I know how to properly use check boxes
- [x] I have included the version of Minecraft I'm running, baritone's version and forge mods (if used).
- [x] I have included logs, exceptions and / or steps to reproduce the issue.
- [x] I have not used any OwO's or UwU's in this issue.

Edit: I am aware of #4110 , the solution does not help on Meteor and happens with just baritone installed.