When I start a build with Litematica, the bot walks to empty places

## Some information
Operating system: Win 10
Java version: 21
Minecraft version: 1.20.6
Baritone version: 1.20.6-20240609.142719-1
Other mods (if used): Meteor Client

## How to reproduce
I use the #litematica command and when it starts to do the build, it walks from the build to an irrelevant place, then it walks to another irrelevant place and so on in an infinite loop. it doesn't do any blocking

## Modified settings
#set allowDiagonalAscend true
#set allowDiagonalDescend true
#set allowInventory true
#set allowParkour true
#set allowParkourPlace true
#set blockBreakSpeed 1
#set blocksToAvoid water
#set blocksToAvoidBreaking crafting_table,furnace,chest,trapped_chest,air
#set buildIgnoreBlocks air,light_gray_concrete,andesite,cyan_terracotta
#set buildIgnoreExisting true
#set buildIgnoreProperties distance,age
#set elytraTermsAccepted true
#set jumpPenalty 4.0
#set layerHeight 2
#set maxYLevelWhileMining 2031
#set rightClickSpeed 2

## Final checklist
- [x] I know how to properly use check boxes
- [x ] I have included the version of Minecraft I'm running, baritone's version and forge mods (if used).
- [x] I have included logs, exceptions and / or steps to reproduce the issue.
- [x ] I have not used any OwO's or UwU's in this issue.