Stuck breaking and replacing the same block during build

Minecraft version: 1.20.1 Fabric
Baritone version: 1.10.1
Other mods: Litematica

## How to reproduce
While using the build command, baritone will repeatedly break and replace a correctly placed block in the schematic. It does this with any type of block, and gets stuck until the command is paused and resumed in a different spot.

Example recording:
https://files.catbox.moe/sj74cq.gif

## Modified settings
BuildInLayers set to true

## Final checklist
- [x] I know how to properly use check boxes
- [x] I have included logs, exceptions and / or steps to reproduce the issue.
- [x] I have included the version of Minecraft I'm running, baritone's version and forge mods (if used).
- [x] I have not used any OwO's or UwU's in this issue.