Quitting neovim takes a long time when toggling git ignore folders

Description

When I toggle a git ignore folders and javascript's node_modules is visible it takes an unusual longer time to quit neovim. I dont mean it only happens with node_modules but tht is my use case. Check the video where I have only nvim-tree as a dependency and I quit neovim when git ignore folders are hidden vs when they are visible.

https://github.com/nvim-tree/nvim-tree.lua/assets/80689446/21ed717f-1b9b-4415-b85c-d9d5d8b31d4d

One weird thing is that if I delete the .gitignore file and the node_modules is visible from the start then it quits just as fast as usual. It is only when I toggle the git ignore files from hidden to visible

https://github.com/nvim-tree/nvim-tree.lua/assets/80689446/5511dec7-511a-499b-aac3-c45370f8fbae

I dont really have any idea why, but let me know if I can help in any way

Neovim version

NVIM v0.9.5
Build type: Release
LuaJIT 2.1.1703358377

Operating system and version

Macos 14.3.1

Windows variant

No response

nvim-tree version

commit: 707b24af91a1d05823099ed8c5450796290466ba

Clean room replication

I tested it without any other plugin

vim.g.loaded_netrw = 1
vim.g.loaded_netrwPlugin = 1

vim.cmd [[set runtimepath=$VIMRUNTIME]]
vim.cmd [[set packpath=/tmp/nvt-min/site]]
local package_root = "/tmp/nvt-min/site/pack"
local install_path = package_root .. "/packer/start/packer.nvim"
local function load_plugins()
  require("packer").startup {
    {
      "wbthomason/packer.nvim",
      "nvim-tree/nvim-tree.lua",
      "nvim-tree/nvim-web-devicons",
      -- ADD PLUGINS THAT ARE _NECESSARY_ FOR REPRODUCING THE ISSUE
    },
    config = {
      package_root = package_root,
      compile_path = install_path .. "/plugin/packer_compiled.lua",
      display = { non_interactive = true },
    },
  }
end
if vim.fn.isdirectory(install_path) == 0 then
  print "Installing nvim-tree and dependencies."
  vim.fn.system { "git", "clone", "--depth=1", "https://github.com/wbthomason/packer.nvim", install_path }
end
load_plugins()
require("packer").sync()
vim.cmd [[autocmd User PackerComplete ++once echo "Ready!" | lua setup()]]
vim.opt.termguicolors = true
vim.opt.cursorline = true

-- MODIFY NVIM-TREE SETTINGS THAT ARE _NECESSARY_ FOR REPRODUCING THE ISSUE
_G.setup = function()
  require("nvim-tree").setup {}
end

-- UNCOMMENT this block for diagnostics issues, substituting pattern and cmd as appropriate.
-- Requires diagnostics.enable = true in setup.
--[[
vim.api.nvim_create_autocmd("FileType", {
  pattern = "lua",
  callback = function()
    vim.lsp.start { cmd = { "lua-language-server" } }
  end,
})
]]

### Steps to reproduce

1. create a directory and cd into it
2. run `npm init -y` to initialize an empty js project
3. run `npm install {add many packages to have a large node_modules}`
4. add a .gitignore with `node_modules`
5. try quitting neovim after toggling gitignore files vs not and see the quitting difference

### Expected behavior

I expect neovim to quit just as fast even when ignored git files are shown

### Actual behavior

Neovim quits slow when hidden git files are shown

Start with the clean-room configuration and reproduction steps using a large node_modules directory, then compare quitting after toggling git-ignored files with quitting when they are visible from startup. Trace nvim-tree's handling of the toggle and shutdown path. Done means Neovim quits at the expected speed in both cases.