Ladders are being used even if facing the other way

## Some information
Operating system: Windows 11
Java version: Java 16
Minecraft version: 1.17.1 (should be an issue with every version)
Baritone version: artifact from pull request #2850
Other mods (if used): uninteresting (no blocks are being modified or anything what baritone should care about)

## How to reproduce
Get map with a block and ladders without a block on the other side (impossible in 1.12.2 without hacking) (the block should be a few blocks at least)
Try to make baritone go up the ladders when the "wrong" side is closer. (baritone always takes the side closer to the start, obviously)

## Modified settings
none

## Video
I know i modified some settings, but i tested it, they don't make a difference
https://youtu.be/gssID17CLKg

## Final checklist
- [x] I know how to properly use check boxes
- [x] I have included the version of Minecraft I'm running, baritone's version and forge mods (if used).
- [x] I have included logs, exceptions and / or steps to reproduce the issue.
- [x] I have not used any OwO's or UwU's in this issue.