Closing fzf and switching back to a terminal will always enter terminal mode

- [X] I have fzf 0.23.0 or above
- [X] I have read through https://github.com/junegunn/fzf.vim/blob/master/README.md
- [X] I have read through https://github.com/junegunn/fzf/blob/master/README-VIM.md
- [X] I have read through the manual page of fzf (`man fzf`)
- [X] I have searched through the existing issues

# Steps to reproduce

1. Setup minimal neovim with fzf
2. Open a `:terminal` without entering terminal mode
3. Run `:FZF`
4. Press escape
5. You are now in the previous terminal in terminal mode

This should not be the case. When switching back to the terminal, it should stay in normal mode.