Bundler.setup raises Bundler::GemNotFound for gems whose native extensions have been removed, even if the gem is already activated

Describe the problem as clearly as you can

This is a problem experienced by some users of the popular puma gem. The actual high-level puma issue is best described in https://github.com/puma/puma/pull/2407 and https://github.com/puma/puma/issues/2018. One possible solution to the puma issue is fixing what I think might be an issue with Bundler.

At a high level, this is what puma does when starting up a cluster-mode server (with option --prune-bundler on):

1. Start puma master process
2. The puma master process executes prune_bundler, which essentially "ejects" the puma master process from the bundler context by execing into a new process and manually activating only the small set of gems on which the puma master process relies directly. The puma master process essentially doesn't use Bundler at all, only using the RubyGems API directly.
3. Worker processes are forked from the puma master process.
4. Worker processes execute require 'bundler/setup' in order to activate the user's application's gems.

The reason why prune_bundler exists at all in puma is to support swapping out the version of the underlying rack application in a process puma calls a "phased restart". A new version of that application might have a different Gemfile.lock and therefore require different versions of gems. Therefore, the puma master process is kept "clean" of the application-specific gems, so that when the master process forks again to support the new version of the application, it can do so without conflicting with its own gems. This strategy mostly works okay. Users can safely update their applications without restarting the puma master process, and do so in a way that allows the server to continue responding to requests during an upgrade.

The problem described in https://github.com/puma/puma/pull/2407 and https://github.com/puma/puma/issues/2018 is that some users have a deployment strategy that automatically removes old versions of their applications (along with the gems associated with those versions). Because of what I think is a quirk of Bundler, this can cause puma phased restarts to fail. Essentially what happens is that users delete the gems that were used when the puma master process first started up (since those gems were associated with an old version of their application). When the puma worker processes execute require 'bundler/setup', they get the exception Bundler::GemNotFound for the gem nio4r. Importantly, nio4r is one of the few gems activated by the puma master process and has native extensions. I traced this error and found that in this case, if the native extensions for a gem are missing on disk, Bundler will throw this error. That of course is desirable for gems that haven't already been activated, but is undesirable for gems that are already activated (the gem is already on the LOAD_PATH and the native extensions are already dynamically loaded into memory, they need not still be on disk).

I know that my understanding of Bundler and RubyGems is not complete, but I think one possible solution is for Bundler.setup to essentially skip the check to ensure that native extensions exist on disk for gems that have already been activated. And of course, I provided all of the background context for puma's issue because we're more than happy to hear any other suggestions that might solve our problem.

Post steps to reproduce the problem

I created a project that reproduces the issue in Docker: https://github.com/cjlarose/bundler-gem-not-found-missing-extensions. In that project, I boiled everything down to only what's necessary, so there's no puma involved. It basically just activates a gem with RubyGems' gem method, deletes the gems from the disk, then executes Bundler.setup.

Just for context, I had previously created a project that produces the higher-level puma issue in Docker: https://github.com/cjlarose/puma-phased-restart-could-not-find-gem-errors

Start with the Bundler.setup entry point, using the Docker reproducer linked in the issue to observe the failure after an activated gem and its files are removed. Done means Bundler.setup no longer raises for an already activated gem whose native extensions are missing, while still checking gems that have not been activated.