`clear_sources` is not honoured when listing available versions from remote

Description

When using a constraint (like ~> 1.0) if a newer gem exists in the default source (e.g. 1.5), but not the source provided in chef_gem (e.g. 1.1) it will try to install the "latest" version (e.g 1.5) against the source in chef_gem resulting in a failure. This occurs even if the following are set:

• clear_sources true
• include_default_source false

It appears that --clear-sources is not passed to candidate_version_from_remote, but it is in install_via_gem_command. This explains why it's not just installing the latest version from the default source, but a mix.

I am not familiar with the codebase, however I've tried to provide a simple fix in: https://github.com/chef/chef/pull/16267

Chef Version

18.11.11

Platform Version

Ubuntu 24.04

Replication Case

Default source is rubygems.org. Use a resource like this:

chef_gem 'cool-gem' do
  action :install
  clear_sources true
  include_default_source false
  source 'https://cool-rubygems.example.com'
  version '~> 1.0'
end

If rubygems.org has 1.5 and cool-rubygems.example.com only has up to 1.2, then the above resource will try to install 1.5 from cool-rubygems.example.com, which fails.

Client Output

I can provide sanitised logs if my investigation is incorrect.

Stacktrace

I can provide a sanitised stacktrace if my investigation is incorrect.

Start in lib/chef/provider/package/rubygems.rb around candidate_version_from_remote and install_via_gem_command, as identified in the issue. Review pull request #16267 and verify the replication case: with clear_sources and include_default_source false, remote version selection should not use versions available only from the default source.