Bundler fails inside rake task

Describe the problem as clearly as you can

Premise: I need to use Ruby 2.7.2 and because of an sqlite3 dependency and the version of sqlite installed on my macbook I ended up with bundler 2.3.18.

In my deployment pipeline I use rake. Some bundler command seem to fail when executed inside a rake task. See below the steps to reproduce.

Did you try upgrading rubygems & bundler?

See Premise.

Post steps to reproduce the problem

Execute the following code

require "fileutils"

desc "Install and deploy (local) arm gems"
task :install_gems_arm do |t|

    puts "Current working directory: #{Dir.pwd}"

    bundle_cmd = "bundle _2.3.18_"

    # Set deployment mode
    puts "Setting deployment mode"
    deployment_output = `#{bundle_cmd} config set --local deployment 'true' 2>&1`
    puts "Output: #{deployment_output}"
    raise "Failed to set deployment mode" unless $?.success?

    # Prepare the local path for gems
    local_path = "vendor/macarm2022/gems"
    puts "Deleting the directory: #{local_path}"
    FileUtils.rm_rf(local_path)
    FileUtils.mkdir_p(local_path) # Ensure the directory exists
    puts "Directory prepared: #{local_path}"

    # Set the bundle path
    puts "Current path: #{Dir.pwd}"
    path_cmd = "#{bundle_cmd} config set --local path \"#{local_path}\""
    puts "Executing: #{path_cmd}"
    path_output = `#{path_cmd} 2>&1`
    puts "Path Command Output: #{path_output}"
    raise "Failed to set bundle path" unless $?.success?
end

with bundle _2.3.18_ exec rake -f bundle.rb install_gems_arm

I get

Current working directory: /Users/ruggiero/dev/MCTools
Setting deployment mode
Output: You are replacing the current local value of deployment, which is currently "false"
Deleting the directory: vendor/macarm2022/gems
Directory prepared: vendor/macarm2022/gems
Current path: /Users/ruggiero/dev/MCTools
Executing: bundle _2.3.18_ config set --local path "vendor/macarm2022/gems"
Path Command Output: /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/definition.rb:498:in `materialize': Could not find aws-sdk-ec2-1.214.0, aws-sdk-s3-1.86.0, clipboard-1.3.5, histogram-0.2.4.1, httparty-0.18.1, net-scp-4.0.0, net-sftp-4.0.0, net-ssh-7.2.1, mqtt-0.6.0, rake-13.2.1, rest-client-2.1.0, ruby_clipper-6.2.1.5.2, rubyzip-2.3.0, sqlite3-1.7.3, color_pound_spec_reporter-0.0.9, minitest-5.25.4, minitest-reporters-1.7.1, minitest-rg-5.3.0, rubocop-1.53.0, rubocop-sketchup-1.5.0, ruby-prof-1.4.2, sketchup-api-stubs-0.7.10, solargraph-0.50.0, yard-0.9.37, aws-sdk-core-3.216.0, aws-sigv4-1.11.0, aws-sdk-kms-1.97.0, mime-types-3.6.0, multi_xml-0.6.0, http-accept-1.7.0, http-cookie-1.0.8, netrc-0.11.0, mini_portile2-2.8.8, ansi-1.5.0, builder-3.3.0, ruby-progressbar-1.13.0, json-2.9.1, language_server-protocol-3.17.0.3, parallel-1.26.3, parser-3.3.7.0, rainbow-3.1.1, regexp_parser-2.10.0, rexml-3.4.0, rubocop-ast-1.37.0, unicode-display_width-2.6.0, backport-1.2.0, benchmark-0.4.0, diff-lcs-1.5.1, e2mmap-0.1.0, jaro_winkler-1.6.0, kramdown-2.5.1, kramdown-parser-gfm-1.1.0, rbs-2.8.4, reverse_markdown-2.1.1, thor-1.3.2, tilt-2.6.0, aws-eventstream-1.3.0, aws-partitions-1.1040.0, jmespath-1.6.2, logger-1.6.5, mime-types-data-3.2025.0107, domain_name-0.6.20240107, ast-2.4.2, racc-1.8.1, nokogiri-1.15.7-arm64-darwin in any of the sources (Bundler::GemNotFound)
        from /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/definition.rb:191:in `specs'
        from /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/definition.rb:247:in `specs_for'
        from /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/runtime.rb:18:in `setup'
        from /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler.rb:162:in `setup'
        from /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/setup.rb:20:in `block in <top (required)>'
        from /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/ui/shell.rb:136:in `with_level'
        from /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/ui/shell.rb:88:in `silence'
        from /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/setup.rb:20:in `<top (required)>'
        from /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
        from /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
rake aborted!
Failed to set bundle path
/Users/ruggiero/dev/MCTools/bundle.rb:29:in `block in <top (required)>'
/Users/ruggiero/.rbenv/versions/2.7.2/bin/bundle:23:in `load'
/Users/ruggiero/.rbenv/versions/2.7.2/bin/bundle:23:in `<main>'
Tasks: TOP => install_gems_arm
(See full trace by running task with --trace)

Now remove the task related lines and execute simply with

ruby bundle.rb

and you get

Current working directory: /Users/ruggiero/dev/MCTools
Setting deployment mode
Output: You are replacing the current local value of deployment, which is currently "false"
Deleting the directory: vendor/macarm2022/gems
Directory prepared: vendor/macarm2022/gems
Current path: /Users/ruggiero/dev/MCTools
Executing: bundle _2.3.18_ config set --local path "vendor/macarm2022/gems"
Path Command Output: 

as expected. What am I missing? Thanks!

If not included with the output of your command, run bundle env and paste the output below

bundle env

Environment

Bundler       2.3.18
  Platforms   ruby, arm64-darwin-21
Ruby          2.7.2p137 (2020-10-01 revision 5445e0435260b449decf2ac16f9d09bae3cafe72) [arm64-darwin-21]
  Full Path   /Users/ruggiero/.rbenv/versions/2.7.2/bin/ruby
  Config Dir  /Users/ruggiero/.rbenv/versions/2.7.2/etc
RubyGems      3.1.4
  Gem Home    /Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0
  Gem Path    /Users/ruggiero/.gem/ruby/2.7.0:/Users/ruggiero/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0
  User Home   /Users/ruggiero
  User Path   /Users/ruggiero/.gem/ruby/2.7.0
  Bin Dir     /Users/ruggiero/.rbenv/versions/2.7.2/bin
Tools         
  Git         2.39.3 (Apple Git-146)
  RVM         not installed
  rbenv       rbenv 1.3.2
  chruby      not installed

Bundler Build Metadata

Built At          2022-07-14
Git SHA           fc31b7f7d9
Released Version  true

Bundler settings

build.libxml-ruby
  Set for the current user (/Users/ruggiero/.bundle/config): "--use-system-libraries --with-xml2-config=/usr/bin/xml2-config"
build.nokogiri
  Set for the current user (/Users/ruggiero/.bundle/config): "--use-system-libraries --with-xml2-lib=/usr/lib --with-xml2-config=/usr/bin/xml2-config"
deployment
  Set for your local app (/Users/ruggiero/dev/MCTools/.bundle/config): true
local
  Set for the current user (/Users/ruggiero/.bundle/config): "set deployment false"
path
  Set for your local app (/Users/ruggiero/dev/MCTools/.bundle/config): "vendor/macarm2022/gems"

Gemfile

Gemfile

# frozen_string_literal: false

source "https://rubygems.org"

ruby "2.7.2"
group :production do
  gem "aws-sdk-ec2", "1.214.0"
  gem "aws-sdk-s3", "1.86.0"
  gem "clipboard", "1.3.5"
  # gem "gsl", "~> 2.1"
  gem "histogram", "0.2.4.1"
  gem "httparty", "0.18.1"
  gem "net-scp", "4.0.0"
  gem "net-sftp", "4.0.0"
  gem "net-ssh", "7.2.1"
  # gem "numo-linalg", "0.1.5"
  # gem "numo-narray", "0.9.1.8"
  gem "mqtt", "0.6.0"
  # gem "mqtt", github: "njh/ruby-mqtt"
  # gem "mqtt", git: "https://github.com/njh/ruby-mqtt.git"
  gem "rake"

  gem "rest-client", "~> 2.1"
  gem "ruby_clipper", "6.2.1.5.2"
  gem "rubyzip", "2.3.0"
  gem "sqlite3", "1.7.3"
end
group :development do
  gem "color_pound_spec_reporter"
  gem "minitest"
  gem "minitest-reporters"
  gem "minitest-rg"
  gem "rubocop", "1.53"
  gem "rubocop-sketchup"
  gem "ruby-prof", "1.4.2"
  gem "sketchup-api-stubs"
  gem "solargraph"
  gem "yard"
end

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    ansi (1.5.0)
    ast (2.4.2)
    aws-eventstream (1.3.0)
    aws-partitions (1.1040.0)
    aws-sdk-core (3.216.0)
      aws-eventstream (~> 1, >= 1.3.0)
      aws-partitions (~> 1, >= 1.992.0)
      aws-sigv4 (~> 1.9)
      jmespath (~> 1, >= 1.6.1)
    aws-sdk-ec2 (1.214.0)
      aws-sdk-core (~> 3, >= 3.109.0)
      aws-sigv4 (~> 1.1)
    aws-sdk-kms (1.97.0)
      aws-sdk-core (~> 3, >= 3.216.0)
      aws-sigv4 (~> 1.5)
    aws-sdk-s3 (1.86.0)
      aws-sdk-core (~> 3, >= 3.109.0)
      aws-sdk-kms (~> 1)
      aws-sigv4 (~> 1.1)
    aws-sigv4 (1.11.0)
      aws-eventstream (~> 1, >= 1.0.2)
    backport (1.2.0)
    benchmark (0.4.0)
    builder (3.3.0)
    clipboard (1.3.5)
    color_pound_spec_reporter (0.0.9)
      bundler
      minitest
      minitest-reporters (>= 0.14.24)
      rake
    diff-lcs (1.5.1)
    domain_name (0.6.20240107)
    e2mmap (0.1.0)
    histogram (0.2.4.1)
    http-accept (1.7.0)
    http-cookie (1.0.8)
      domain_name (~> 0.5)
    httparty (0.18.1)
      mime-types (~> 3.0)
      multi_xml (>= 0.5.2)
    jaro_winkler (1.6.0)
    jmespath (1.6.2)
    json (2.9.1)
    kramdown (2.5.1)
      rexml (>= 3.3.9)
    kramdown-parser-gfm (1.1.0)
      kramdown (~> 2.0)
    language_server-protocol (3.17.0.3)
    logger (1.6.5)
    mime-types (3.6.0)
      logger
      mime-types-data (~> 3.2015)
    mime-types-data (3.2025.0107)
    mini_portile2 (2.8.8)
    minitest (5.25.4)
    minitest-reporters (1.7.1)
      ansi
      builder
      minitest (>= 5.0)
      ruby-progressbar
    minitest-rg (5.3.0)
      minitest (~> 5.0)
    mqtt (0.6.0)
    multi_xml (0.6.0)
    net-scp (4.0.0)
      net-ssh (>= 2.6.5, < 8.0.0)
    net-sftp (4.0.0)
      net-ssh (>= 5.0.0, < 8.0.0)
    net-ssh (7.2.1)
    netrc (0.11.0)
    nokogiri (1.15.7-arm64-darwin)
      racc (~> 1.4)
    parallel (1.26.3)
    parser (3.3.7.0)
      ast (~> 2.4.1)
      racc
    racc (1.8.1)
    rainbow (3.1.1)
    rake (13.2.1)
    rbs (2.8.4)
    regexp_parser (2.10.0)
    rest-client (2.1.0)
      http-accept (>= 1.7.0, < 2.0)
      http-cookie (>= 1.0.2, < 2.0)
      mime-types (>= 1.16, < 4.0)
      netrc (~> 0.8)
    reverse_markdown (2.1.1)
      nokogiri
    rexml (3.4.0)
    rubocop (1.53.0)
      json (~> 2.3)
      language_server-protocol (>= 3.17.0)
      parallel (~> 1.10)
      parser (>= 3.2.2.3)
      rainbow (>= 2.2.2, < 4.0)
      regexp_parser (>= 1.8, < 3.0)
      rexml (>= 3.2.5, < 4.0)
      rubocop-ast (>= 1.28.0, < 2.0)
      ruby-progressbar (~> 1.7)
      unicode-display_width (>= 2.4.0, < 3.0)
    rubocop-ast (1.37.0)
      parser (>= 3.3.1.0)
    rubocop-sketchup (1.5.0)
      rubocop (>= 0.82, < 2.0)
    ruby-prof (1.4.2)
    ruby-progressbar (1.13.0)
    ruby_clipper (6.2.1.5.2)
    rubyzip (2.3.0)
    sketchup-api-stubs (0.7.10)
    solargraph (0.50.0)
      backport (~> 1.2)
      benchmark
      bundler (~> 2.0)
      diff-lcs (~> 1.4)
      e2mmap
      jaro_winkler (~> 1.5)
      kramdown (~> 2.3)
      kramdown-parser-gfm (~> 1.1)
      parser (~> 3.0)
      rbs (~> 2.0)
      reverse_markdown (~> 2.0)
      rubocop (~> 1.38)
      thor (~> 1.0)
      tilt (~> 2.0)
      yard (~> 0.9, >= 0.9.24)
    sqlite3 (1.7.3)
      mini_portile2 (~> 2.8.0)
    thor (1.3.2)
    tilt (2.6.0)
    unicode-display_width (2.6.0)
    yard (0.9.37)

PLATFORMS
  arm64-darwin-21

DEPENDENCIES
  aws-sdk-ec2 (= 1.214.0)
  aws-sdk-s3 (= 1.86.0)
  clipboard (= 1.3.5)
  color_pound_spec_reporter
  histogram (= 0.2.4.1)
  httparty (= 0.18.1)
  minitest
  minitest-reporters
  minitest-rg
  mqtt (= 0.6.0)
  net-scp (= 4.0.0)
  net-sftp (= 4.0.0)
  net-ssh (= 7.2.1)
  rake
  rest-client (~> 2.1)
  rubocop (= 1.53)
  rubocop-sketchup
  ruby-prof (= 1.4.2)
  ruby_clipper (= 6.2.1.5.2)
  rubyzip (= 2.3.0)
  sketchup-api-stubs
  solargraph
  sqlite3 (= 1.7.3)
  yard

RUBY VERSION
   ruby 2.7.2p137

BUNDLED WITH
   2.3.18

Start with the reproduced commands in bundle.rb, comparing bundle _2.3.18_ exec rake -f bundle.rb install_gems_arm with ruby bundle.rb. Read the Bundler setup path involved in the reported stack trace and use the Gemfile and Gemfile.lock to reproduce the missing-gem state. Done means the rake-task invocation no longer fails in the reported scenario, with coverage for the regression.