"Key not present in Dictionary" error in every fzf function in one specific project folder with neovim

- [x] I have fzf 0.22.0 or above
- [x] I have read through https://github.com/junegunn/fzf.vim/blob/master/README.md
- [x] I have read through https://github.com/junegunn/fzf/blob/master/README-VIM.md
- [ ] I have read through the manual page of fzf (`man fzf`)
- [x] I have searched through the existing issues

I have one specific project folder where every fzf function (e.g. `:Files`, `:GFiles`) shows this error and opens an empty popup afterwards:

>Error detected while processing function fzf#vim#files[15]..172_fzf[18]..fzf#run[58]..42_execute_term[82]..42_dopopd:
>line 20:
>E716: Key not present in Dictionary: dir && (!&autochdir || w:fzf_pushd.bufname ==# bufname(''))
>E15: Invalid expression: s:fzf_getcwd() ==# w:fzf_pushd.dir && (!&autochdir || w:fzf_pushd.bufname ==# bufname(''))

I'm on https://github.com/junegunn/fzf/commit/c0aa5a438f2484c9f931d1258e2eeb5ddc5927ad and https://github.com/junegunn/fzf.vim/commit/0fe8e198a3a501b54dbc4f9587526c097599f95a.

This also happens when running the minimal configuration in neovim, but not in vim.

Neovim version:

```
$ LANG=C nvim --version
NVIM v0.4.4
Build type: Release
LuaJIT 2.0.5
Compilation: /usr/bin/clang -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNDEBUG -DMIN_LOG_LEVEL=3 -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -I/tmp/neovim-20200808-89283-1us0ytm/neovim-0.4.4/build/config -I/tmp/neovim-20200808-89283-1us0ytm/neovim-0.4.4/src -I/usr/local/include -I/tmp/neovim-20200808-89283-1us0ytm/neovim-0.4.4/deps-build/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -I/usr/local/opt/gettext/include -I/tmp/neovim-20200808-89283-1us0ytm/neovim-0.4.4/build/src/nvim/auto -I/tmp/neovim-20200808-89283-1us0ytm/neovim-0.4.4/build/include
Compiled by brew@Catalina

Features: +acl +iconv +tui
See ":help feature-compile"

system vimrc file: "$VIM/sysinit.vim"
fall-back for $VIM: "/usr/local/Cellar/neovim/0.4.4/share/nvim"

Run :checkhealth for more info
```

I also tried the most recent neovim master build, but it happens there too.

#689 could be related, but none of the workarounds there worked for me.