Map-specific Portraits.xml stays loaded after exiting map

### My mods and Everest install are up to date

Yes

### I have recreated the bug with only Everest OR a minimum number of mods enabled

Yes

### Describe the bug

A map-specific Portraits.xml stays loaded after exiting map, leading to issues when portraits are used outside of maps, e.g. on the save file screen.

### Steps to reproduce

1. Enter a map with a custom Portraits.xml that changes the Madeline portrait.
2. Exit the map.
3. Open the save file screen. The portrait will be changed, or the game will crash if the xml has no Madeline portrait.

### Expected behavior

When exiting the map, the game should switch back to the global xml.

### Operating System

all

### Everest Version

4986

### Mods required to reproduce

_No response_

### Additional context

_No response_