ruby / ruby/rubygems

Memory exhaustion on AIX 7.2 on Rubygems >= 4.0

Open
#9,368 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

RubyGems
Dominant language
Ruby
Stars
4k
Forks
1.9k
Avg merge
1d 2h
Merged PRs (30d)
81

Description

Describe the problem as clearly as you can

The SafeMarshal feature of Rubygems >= 4.0.0 causes memory exhaustion on AIX 7.2 when adding sources.

With Rubygems 4.0.3 on Ruby 4.0.1:

[0] [AIX] root@aix72-2-pix:~ # /opt/puppetlabs/puppet/bin/gem source --add $internal_artifactory_repo
[FATAL] failed to allocate memory
Did you try upgrading RubyGems?

This issue also seems to affect RubyGems upgrades:

[0] [AIX] root@aix72-2-pix:~ # /opt/puppetlabs/puppet/bin/gem update --system
[FATAL] failed to allocate memory
Post steps to reproduce the problem

The simplest reproduction is to take an AIX 7.2 machine and attempt to gem source --add.

@joshcooper also created this reproduction script:

require 'rubygems'
require 'rubygems/safe_marshal'
require 'objspace'

def safe_marshal_element_classes
  result = []
  Gem::SafeMarshal::Elements.constants(false).each do |const|
    c = Gem::SafeMarshal::Elements.const_get(const)
    result << c if c.is_a?(Class)
  rescue NameError
    # ignore
  end
  result
end

def histogram_counts(limit: 50, min_count: 1)
  klasses = safe_marshal_element_classes
  counts  = klasses.map do |k|
    n = 0
    ObjectSpace.each_object(k) { n += 1 }
    [k, n]
  end

  counts.select! { |(_, n)| n >= min_count }
  counts.sort_by! { |(_, n)| -n }
  counts.first(limit)
end

io = Gem.read_binary(ARGV[0])
data = Gem::SafeMarshal.safe_load(io)
#tuples = Gem::NameTuple.from_list(rval)
histogram_counts(limit: 25).each do |klass, n|
  puts "#{klass}: #{n}"
end

Even when attempting to load specs from rubygems.org, we see the same issue:

[0] [AIX] root@aix72-2-pix:~ # curl -s --output rubygems-specs.4.8.gz https://rubygems.org/specs.4.8.gz
[0] [AIX] root@aix72-2-pix:~ # gunzip rubygems-specs.4.8.gz
[0] [AIX] root@aix72-2-pix:~ # /opt/puppetlabs/puppet/bin/ruby spec_histogram.rb rubygems-specs.4.8
[FATAL] failed to allocate memory
Which command did you run?
gem source --add
What were you expecting to happen?

Rubygems adds another source for gems.

What actually happened?

The command failed with [FATAL] failed to allocate memory.

Run gem env and paste the output below
[0] [AIX] root@aix72-2-pix:~ # /opt/puppetlabs/puppet/bin/gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 4.0.3
  - RUBY VERSION: 4.0.1 (2026-01-13 patchlevel 0) [powerpc-aix7.2.0.0]
  - INSTALLATION DIRECTORY: /opt/puppetlabs/puppet/lib/ruby/gems/4.0.0
  - USER INSTALLATION DIRECTORY: /root/.local/share/gem/ruby/4.0.0
  - CREDENTIALS FILE: /root/.local/share/gem/credentials
  - RUBY EXECUTABLE: /opt/puppetlabs/puppet/bin/ruby
  - GIT EXECUTABLE:
  - EXECUTABLE DIRECTORY: /opt/puppetlabs/puppet/bin
  - SPEC CACHE DIRECTORY: /root/.cache/gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /opt/puppetlabs/puppet/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - powerpc-aix-7
  - GEM PATHS:
     - /opt/puppetlabs/puppet/lib/ruby/gems/4.0.0
     - /root/.local/share/gem/ruby/4.0.0
     - /opt/puppetlabs/puppet/lib/ruby/vendor_gems
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     -
     -
     - /usr/bin
     - /etc
     - /usr/sbin

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the rubygems/safe_marshal entry point and reproduce the failure using gem source --add or the provided spec_histogram.rb script on AIX 7.2. Compare SafeMarshal.safe_load behavior and memory usage for the supplied specs; done means these operations no longer exhaust memory on the reported platform.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
devtools, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.