Could boot performance be improved? Caching default gems perhaps?

Sorry if this was discussed elsewhere, I tried to search for previous discussions but given how generic the subject is I couldn't find anything.

Context

As many people know, Rubygems initialization is the biggest part of MRI's boot:

$ time ruby -e '1'

real	0m0.076s
user	0m0.051s
sys	0m0.021s
$ time ruby --disable-gems -e '1'

real	0m0.029s
user	0m0.011s
sys	0m0.015s

So (on my machine) "raw" MRI initialization is 29ms, and rubygems adds 47ms to it.

When profiling, we can see that a large part of that comes from loading default gems:

==================================
  Mode: cpu(10)
  Samples: 545 (52.98% miss rate)
  GC: 25 (4.59%)
==================================
     TOTAL    (pct)     SAMPLES    (pct)     FRAME
       237  (43.5%)         190  (34.9%)     Gem::Specification.load
       419  (76.9%)          40   (7.3%)     <require:/opt/rubies/2.8.0-dev/lib/ruby/2.8.0/rubygems.rb>
        66  (12.1%)          34   (6.2%)     <require:/opt/rubies/2.8.0-dev/lib/ruby/2.8.0/did_you_mean.rb>
        70  (12.8%)          33   (6.1%)     <require:/opt/rubies/2.8.0-dev/lib/ruby/2.8.0/rubygems/specification.rb>
        30   (5.5%)          30   (5.5%)     Gem::BasicSpecification#internal_init
        22   (4.0%)          22   (4.0%)     (sweeping)
        93  (17.1%)          21   (3.9%)     <module:Gem>
       106  (19.4%)          16   (2.9%)     Kernel#require
        15   (2.8%)          14   (2.6%)     Gem::Specification#files
        10   (1.8%)          10   (1.8%)     Gem::PathSupport#expand
        11   (2.0%)           9   (1.7%)     <require:/opt/rubies/2.8.0-dev/lib/ruby/2.8.0/did_you_mean/spell_checker.rb>
        10   (1.8%)           9   (1.7%)     <require:/opt/rubies/2.8.0-dev/lib/ruby/2.8.0/rubygems/user_interaction.rb>
        11   (2.0%)           9   (1.7%)     <require:/opt/rubies/2.8.0-dev/lib/ruby/2.8.0/did_you_mean/spell_checkers/name_error_checkers.rb>
         8   (1.5%)           8   (1.5%)     Dir.glob
        14   (2.6%)           7   (1.3%)     Gem::Specification#add_dependency_with_type
        17   (3.1%)           7   (1.3%)     <require:/opt/rubies/2.8.0-dev/lib/ruby/2.8.0/rubygems/specification_policy.rb>
        23   (4.2%)           5   (0.9%)     Gem.register_default_spec
        10   (1.8%)           4   (0.7%)     <class:Specification>
         5   (0.9%)           4   (0.7%)     <require:/opt/rubies/2.8.0-dev/lib/ruby/2.8.0/rubygems/core_ext/kernel_require.rb>
         5   (0.9%)           4   (0.7%)     <require:/opt/rubies/2.8.0-dev/lib/ruby/2.8.0/did_you_mean/spell_checkers/require_path_checker.rb>
         5   (0.9%)           4   (0.7%)     <require:/opt/rubies/2.8.0-dev/lib/ruby/2.8.0/rubygems/dependency.rb>

If I comment out Gem::Specification.load_defaults, the overhead is down to 26ms:

$ time ruby -e '1'

real	0m0.058s
user	0m0.034s
sys	0m0.020s
$ time ruby --disable-gems  -e '1'

real	0m0.032s
user	0m0.012s
sys	0m0.016s

And most of that is in each_spec([Gem.default_specifications_dir]).

Ideas

This makes me wonder what kind of assumptions we could make about default gems. Can we assume they are immutable? If so couldn't we cache all these specs in a single marshal dump file rather than evaluate all the default gems (55 on 2.8.0-dev) ?

I tried a very crude hack on my machine:

  def self.load_defaults
    cached_specs.each do |spec|
      # #load returns nil if the spec is bad, so we just ignore
      # it at this stage
      Gem.register_default_spec(spec)
    end
  end

  def self.cached_specs
    Marshal.load(File.read('/tmp/default_specs.marshal'))
  rescue SystemCallError
    specs = []
    each_spec([Gem.default_specifications_dir]) { |s| specs << s }
    File.write('/tmp/default_specs.marshal', Marshal.dump(specs))
    specs
  end

With the following result:

$ time ruby -e '1'

real	0m0.060s
user	0m0.036s
sys	0m0.020s

So 16ms faster, which is about 34% of rubygems load time.

Questions

So the main question here is what would it take to move this from a crude hack to an acceptable patch.

• Can we assume default gems are immutable?
• If not could we do a mtime comparison? on the directory or all individual specs?
• Is there an existing location where it would be acceptable to store that cache?

@deivid-rodriguez any thoughts on this?

Start with Rubygems initialization, especially Gem::Specification.load_defaults, each_spec([Gem.default_specifications_dir]), and Gem.register_default_spec. Reproduce the reported ruby -e and --disable-gems timings, then evaluate the proposed Marshal cache and its immutability or mtime questions. Done means an accepted approach that improves default-gem boot time without breaking specification loading.