Baritone endless pathing when building schematics.

## Some information
Operating system: macOS Big Sur 11.3.1 (M1 chip)
Java version: Java 8 Update 291
Minecraft version: 1.16.5
Baritone version: v1.6.3
Other mods (if used): Sodium, Lithium, Phosphor, BleachHack, Litematica, WorldEdit, Tweakeroo, MiniHUD, malilib

## Exception, error or logs
[latest.log](https://github.com/cabaletta/baritone/files/6835492/latest.log)

## How to reproduce
1. build a schematic
2. Turn of BuildInLayers (bot builds it without an issue)
3. Turn on BuildInLayers
4. Bot started to stand there and pathing forever.

please watch this video:

https://user-images.githubusercontent.com/56954729/126052562-e1b3b55a-ec0d-48c6-b011-4c73d36fd582.mp4

## Modified settings
All the modified settings are shown in the video
But here's a list anyway:
AllowInventory True
BlockPlacementPenalty 20.0
BuildIgnoreBlocks air
BuildIgnoreExisting true
RightClickSpeed 2

## Final checklist
- [x] I know how to properly use check boxes
- [x] I have included the version of Minecraft I'm running, baritone's version and forge mods (if used).
- [x] I have included logs, exceptions and / or steps to reproduce the issue.
- [x] I have not used any OwO's or UwU's in this issue.