'conflicting chdir during another chdir block' calling vite_asset_path

[troubleshooting section]: https://vite-ruby.netlify.app/guide/troubleshooting.html

- [ ] I have tried upgrading by running `bundle update vite_ruby`.
- [x] I have read the __[troubleshooting section]__ before opening an issue.

### Description 📖

I have a page with a number of iframes loading PDF previews. When loading, some of them fail with:

```
RuntimeError: conflicting chdir during another chdir block (RuntimeError)
from vite_ruby (3.3.4) lib/vite_ruby/config.rb:84:in `chdir'
from vite_ruby (3.3.4) lib/vite_ruby/config.rb:84:in `within_root'
from vite_ruby (3.3.4) lib/vite_ruby/builder.rb:57:in `watched_files_digest'
from vite_ruby (3.3.4) lib/vite_ruby/builder.rb:32:in `last_build_metadata'
from vite_ruby (3.3.4) lib/vite_ruby/manifest.rb:217:in `missing_entry_error'
from vite_ruby (3.3.4) lib/vite_ruby/manifest.rb:79:in `lookup!'
from vite_ruby (3.3.4) lib/vite_ruby/manifest.rb:22:in `path_for'
from vite_rails (3.0.17) lib/vite_rails/tag_helpers.rb:26:in `vite_asset_path'
```

When loading them individually, they work. This seems like a thread safety issue, as [chdir is not thread-safe](https://bugs.ruby-lang.org/issues/15661).

Vite Ruby Info

```
bin/vite present?: true
vite_ruby: 3.3.4
vite_rails: 3.0.17
rails: 7.0.8
node: v20.11.0
npm: 10.2.4
yarn: 1.22.5
pnpm: 7.14.2
ruby: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
```