Can't path while immersive-portals is installed (forge 1.16.4)

## Some information
Operating system: Windows 10
Java version: 1.8
Minecraft version: 1.16.4
Baritone version: 1.6.2
Forge mods (if used): [Immersive Portals](https://www.curseforge.com/minecraft/mc-mods/immersive-portals-for-forge)

## Exception, error or logs
[latest.log](https://paste.ee/p/PXBaz)

## How to reproduce
Add your steps to reproduce the issue/bug experienced here.
1. Install immersive-portals alongside baritone-api-forge
2. Create a new world
3. Attempt to path somewhere (`#thisway 100`, `#path`)
4. Notice nothing happens and with `debugChat` enabled get spammed with `Pausing since destination is at edge of loaded chunks`

## Modified settings
To get the modified settings run `#modified` in game
Only `debugChat true`

## Final checklist
- [x] I know how to properly use check boxes
- [x] I have included the version of Minecraft I'm running, baritone's version and forge mods (if used).
- [x] I have included logs, exceptions and / or steps to reproduce the issue.
- [x] I have not used any OwO's or UwU's in this issue.