auto_install fails if Gemfile.lock has stdlib gem dependency that bundler also uses

Describe the problem as clearly as you can

While working on https://github.com/rubygems/rubygems/pull/6561 , I found that if I try to use a bundle subcommand that supports auto_install, and the bundle contains a version of a stdlib that bundler uses but is a different version from what is in the stdlib, then the command will successfully auto_install, but then fail to actually run because a different version of the gem has been activated.

Subsequent calls do work.

Did you try upgrading rubygems & bundler?

Yes, tested on rubygems 3.4.18 and bundler 2.4.18.

I've also bundler tested on master while working on https://github.com/rubygems/rubygems/pull/6561

Post steps to reproduce the problem

Create a Gemfile:

# frozen_string_literal: true

source "https://rubygems.org"

# could be anything, just happened to choose this
gem "rubocop" 

# needs to be something from stdlib that bundler uses
# and a version different from what's in the standard gems
#
# see https://stdgems.org/ for current gems & version
# ruby 3.2.2 uses 5.0.1, for example
gem "psych", "4.0.3"

Run the commands:

bundle config auto_install true

# make sure the gem is missing, so auto_install will kick in
gem uninstall rubocop --force --executables

# run something that triggers auto install, ie `bundle exec`

Which command did you run?

Run a bundle subcommand that uses auto_install (ie exec):

$ bundle exec rubocop --version

What were you expecting to happen?

I'd have expected the gems to be installed, and then run the command, ie:

$ bundle exec rubocop --version
1.56.0

What actually happened?

$ bundle exec rubocop --version
Automatically installing missing gems.
Fetching gem metadata from https://rubygems.org/........
Fetching rubocop 1.56.0
Installing rubocop 1.56.0
Bundle complete! 2 Gemfile dependencies, 17 gems now installed.
Use bundle info [gemname] to see where a bundled gem is installed.
bundler: failed to load command: rubocop (/Users/josh.nichols/.rbenv/versions/3.2.2/bin/rubocop)
/Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/runtime.rb:304:in check_for_activated_spec!': You have already activated psych 5.1.0, but your Gemfile requires psych 4.0.3. Prepending bundle exec to your command may solve this. (Gem::LoadError)
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/runtime.rb:25:in block in setup'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/spec_set.rb:165:in each'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/spec_set.rb:165:in each'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/runtime.rb:24:in map'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/runtime.rb:24:in setup'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler.rb:162:in setup'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/setup.rb:10:in block in <top (required)>'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/ui/shell.rb:159:in with_level'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/ui/shell.rb:111:in silence'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/setup.rb:10:in <top (required)>'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:56:in require_relative'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:56:in kernel_load'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:23:in run'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/cli.rb:492:in exec'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor.rb:392:in dispatch'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/cli.rb:34:in dispatch'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/base.rb:485:in start'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/cli.rb:28:in start'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/exe/bundle:37:in block in <top (required)>'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/lib/bundler/friendly_errors.rb:117:in with_friendly_errors'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.4.18/exe/bundle:29:in <top (required)>'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/bin/bundle:25:in load'
	from /Users/josh.nichols/.rbenv/versions/3.2.2/bin/bundle:25:in <main>'

Environment

Bundler       2.4.18
  Platforms   ruby, arm64-darwin-22
Ruby          3.2.2p53 (2023-03-30 revision e51014f9c05aa65cbf203442d37fef7c12390015) [arm64-darwin-22]
  Full Path   /Users/josh.nichols/.rbenv/versions/3.2.2/bin/ruby
  Config Dir  /Users/josh.nichols/.rbenv/versions/3.2.2/etc
RubyGems      3.4.12
  Gem Home    /Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0
  Gem Path    /Users/josh.nichols/.gem/ruby/3.2.0:/Users/josh.nichols/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0
  User Home   /Users/josh.nichols
  User Path   /Users/josh.nichols/.gem/ruby/3.2.0
  Bin Dir     /Users/josh.nichols/.rbenv/versions/3.2.2/bin
Tools
  Git         2.41.0
  RVM         not installed
  rbenv       rbenv 1.2.0
  chruby

Bundler Build Metadata

Built At          2023-08-02
Git SHA           d2e3d8e3f4
Released Version  true

Bundler settings

auto_install
  Set for your local app (/Users/josh.nichols/workspace/auto-install-demo/.bundle/config): true
build.idn-ruby
  Set for the current user (/Users/josh.nichols/.bundle/config): "--with-idn-dir=/opt/homebrew/opt/libidn"
gem.changelog
  Set for the current user (/Users/josh.nichols/.bundle/config): false
gem.ci
  Set for the current user (/Users/josh.nichols/.bundle/config): false
gem.coc
  Set for the current user (/Users/josh.nichols/.bundle/config): false
gem.linter
  Set for the current user (/Users/josh.nichols/.bundle/config): false
gem.mit
  Set for the current user (/Users/josh.nichols/.bundle/config): false
gem.test
  Set for the current user (/Users/josh.nichols/.bundle/config): false
unset
  Set for the current user (/Users/josh.nichols/.bundle/config): "jobs"

Gemfile

Gemfile

# frozen_string_literal: true

source "https://rubygems.org"

gem "rubocop"
gem "psych", "4.0.3"

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    ast (2.4.2)
    base64 (0.1.1)
    json (2.6.3)
    language_server-protocol (3.17.0.3)
    parallel (1.23.0)
    parser (3.2.2.3)
      ast (~> 2.4.1)
      racc
    psych (4.0.3)
      stringio
    racc (1.7.1)
    rainbow (3.1.1)
    regexp_parser (2.8.1)
    rexml (3.2.6)
    rubocop (1.56.0)
      base64 (~> 0.1.1)
      json (~> 2.3)
      language_server-protocol (>= 3.17.0)
      parallel (~> 1.10)
      parser (>= 3.2.2.3)
      rainbow (>= 2.2.2, < 4.0)
      regexp_parser (>= 1.8, < 3.0)
      rexml (>= 3.2.5, < 4.0)
      rubocop-ast (>= 1.28.1, < 2.0)
      ruby-progressbar (~> 1.7)
      unicode-display_width (>= 2.4.0, < 3.0)
    rubocop-ast (1.29.0)
      parser (>= 3.2.1.0)
    ruby-progressbar (1.13.0)
    stringio (3.0.8)
    unicode-display_width (2.4.2)

PLATFORMS
  arm64-darwin-22

DEPENDENCIES
  psych (= 4.0.3)
  rubocop

BUNDLED WITH
   2.4.18

Start with the bundle exec entry point and the auto_install configuration described in the issue, reproducing the failure with the provided Gemfile and Gemfile.lock using psych 4.0.3. Trace how installation and runtime activation interact; done means the first bundle exec rubocop --version succeeds without the activated psych version conflict.