Incompatible gems in optional groups is more difficult since Bundler 2.4.21

Describe the problem as clearly as you can

I have a Gemfile that looks like this: https://github.com/Homebrew/brew/blob/master/Library/Homebrew/Gemfile. Ignore the Dependabot-specific hack at the top. The lockfile is here: https://github.com/Homebrew/brew/blob/master/Library/Homebrew/Gemfile.lock.

The basic scenario is I have:

group :typecheck, optional: true do
  gem "sorbet-static-and-runtime", require: false
end

and in the lockfile:

...
    sorbet-static (0.5.10461-universal-darwin-14)
    sorbet-static (0.5.10461-universal-darwin-15)
    sorbet-static (0.5.10461-universal-darwin-16)
    sorbet-static (0.5.10461-universal-darwin-17)
    sorbet-static (0.5.10461-universal-darwin-18)
    sorbet-static (0.5.10461-universal-darwin-19)
    sorbet-static (0.5.10461-universal-darwin-20)
    sorbet-static (0.5.10461-universal-darwin-21)
    sorbet-static (0.5.10461-universal-darwin-22)
    sorbet-static (0.5.10461-x86_64-linux)
...

PLATFORMS
  aarch64-linux
  arm-linux
  arm64-darwin
  x86_64-darwin
  x86_64-linux

This is perhaps unconventional as technically speaking aarch64-linux and arm-linux are invalid for sorbet-static as there is no compatible gem. And indeed if you are on those platforms and try to run bundle update/bundle lock --update or try to install the typecheck group, you will indeed get resolver issues.

However, sorbet-static is in an optional group and bundle install without the group works perfectly fine on arm64 Linux. For us the group of people modifying and updating the Gemfile and the people installing from the Gemfile are two distinct groups. The latter group doesn't even need to know about the concept of a group: we pass the relevant groups as necessary. Notably, it is acceptable for us to be unable to maintain the Gemfile on arm64 Linux but still be able to install from it.

This all however worked ok until today we saw Dependabot open a PR that started removing one of the bad platforms: https://github.com/Homebrew/brew/pull/16225/files (I'll send a PR that fixes it not removing both platforms). This was because Dependabot now uses Bundler 2.4.22 and it seems 2.4.21 changed the behaviour to start force removing invalid platforms (https://github.com/rubygems/rubygems/pull/7035).

The behaviour change makes sense, but for specifically the use case of optional groups it doesn't make sense to me why the user should be blocked from installing everything entirely (since Bundler force removing the platform will mean the next bundle install will block unrecognised platforms) because of one optional group they are not requesting. It is also not particularly different than say the Darwin 23 scenario, which is also not technically valid with the above lockfile in a similar way, but Bundler doesn't currently force x86_64-darwin to become x86_64-darwin-14 through 22.

Any thoughts on perhaps not checking gems that are in optional groups (and maybe install_if blocks)? Or otherwise other paths to achieve the previous behaviour?

(I recognise it's a complicated issue. Longer term maybe it would make sense for new DSL similar to https://bundler.io/v2.4/man/gemfile.5.html#PLATFORMS to conditionally apply gems to certain platforms and for bundle lock --update to behave agnostically to the current running platform. That's for another issue however.)

Did you try upgrading rubygems & bundler?

The issue regressed in Bundler 2.4.21 and is present still in 2.4.22.

Post steps to reproduce the problem

Create a gemfile with sorbet-static in an optional group, resolve the lockfile and add bundle lock --add-platform aarch64-linux.
Note that it on 2.4.20 it installs correctly on arm64 Linux.
Then note that any non-frozen operation on the lockfile since 2.4.21 will remove the platform.

Which command did you run?

bundle lock --update (and whatever Dependabot uses)

What were you expecting to happen?

The lockfile is usable and left untouched when incompatible gems are kept in optional groups.

What actually happened?

The platforms are removed and the entire lockfile is now unusable on said platforms.

If not included with the output of your command, run bundle env and paste the output below

Bundler       2.4.18
  Platforms   ruby, x86_64-darwin-15
Ruby          3.1.4p223 (2023-03-30 revision 957bb7cb81995f26c671afce0ee50a5c660e540e) [x86_64-darwin-15]
  Full Path   /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
  Config Dir  /usr/local/Cellar/portable-ruby/3.1.4/etc
RubyGems      3.3.26
  Gem Home    /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.1.0
  Gem Path    /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.1.0
  User Home   /Users/bo
  User Path   /Users/bo/.gem/ruby/3.1.0
  Bin Dir     /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.1.0/bin
Tools         
  Git         2.37.1 (Apple Git-137.1)
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Built At          2023-08-02
Git SHA           d2e3d8e3f4
Released Version  true

bin
  Set for your local app (/usr/local/Homebrew/Library/Homebrew/.bundle/config): false
clean
  Set for your local app (/usr/local/Homebrew/Library/Homebrew/.bundle/config): true
disable_shared_gems
  Set for your local app (/usr/local/Homebrew/Library/Homebrew/.bundle/config): true
force_ruby_platform
  Set for your local app (/usr/local/Homebrew/Library/Homebrew/.bundle/config): false
forget_cli_options
  Set for your local app (/usr/local/Homebrew/Library/Homebrew/.bundle/config): true
gem.ci
  Set for the current user (/Users/bo/.bundle/config): "github"
gem.linter
  Set for the current user (/Users/bo/.bundle/config): "rubocop"
gem.test
  Set for the current user (/Users/bo/.bundle/config): "rspec"
gemfile
  Set via BUNDLE_GEMFILE: "/usr/local/Homebrew/Library/Homebrew/Gemfile"
jobs
  Set for your local app (/usr/local/Homebrew/Library/Homebrew/.bundle/config): 4
path
  Set for your local app (/usr/local/Homebrew/Library/Homebrew/.bundle/config): "vendor/bundle"
retry
  Set for your local app (/usr/local/Homebrew/Library/Homebrew/.bundle/config): 3

Start with the linked Homebrew Gemfile and Gemfile.lock, then reproduce the behavior with bundle lock --update and bundle lock --add-platform aarch64-linux across Bundler 2.4.20 and 2.4.21. Trace the platform removal for the optional typecheck group and compare it with the change discussed in rubygems/rubygems#7035. Done means optional incompatible gems no longer make the lockfile unusable when that group is not requested.