Installing native extensions at the same time

Describe the problem as clearly as you can

When installing gems with native extensions that look for headers, if you have two processes installing the same gem at the same time, one of the processes will fail. My current hypothesis (which I think/hope is correct) is that one copy of the gem is being extracted on top of another, and so the process compiling the first native extension no longer finds the headers that it needs.

This was reported as a bug with prism (https://github.com/ruby/prism/issues/2868) but I think the issue is at the rubygems level, because I was able to reproduce with multiple gems that have native extensions.

Did you try upgrading RubyGems?

Yes, I am on latest.

Post steps to reproduce the problem

Open two terminal windows and run gem install [name] at the same time for any gem that looks for headers. I replicated this issue with prism, ffi, and sqlite3, but I'm sure it applies to others as well.

Which command did you run?

See above.

What were you expecting to happen?

I expect under the hood that it locks the directory through something like flock or similar in order to prevent this from happening.

What actually happened?

One process fails to install.

Run gem env and paste the output below

RubyGems Environment:
  - RUBYGEMS VERSION: 3.6.0.dev
  - RUBY VERSION: 3.4.0 (2024-02-07 patchlevel -1) [arm64-darwin23]
  - INSTALLATION DIRECTORY: ~/.gem/ruby/3.4.0
  - USER INSTALLATION DIRECTORY: ~/.gem/ruby/3.4.0+0
  - RUBY EXECUTABLE: ~/.rubies/ruby-3.4.0-dev/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: ~/.gem/ruby/3.4.0/bin
  - SPEC CACHE DIRECTORY: ~/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: ~/.rubies/ruby-3.4.0-dev/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - arm64-darwin-23
  - GEM PATHS:
     - ~/.gem/ruby/3.4.0
     - ~/.rubies/ruby-3.4.0-dev/lib/ruby/gems/3.4.0+0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - :sources => ["https://rubygems.org/"]
     - :install => "--no-document"
     - :update => "--user --no-document"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - ~/.gem/ruby/3.4.0/bin
     - ~/.rubies/ruby-3.4.0-dev/lib/ruby/gems/3.4.0+0/bin
     - ~/.rubies/ruby-3.4.0-dev/bin
     - /opt/homebrew/bin
     - /opt/homebrew/sbin
     - /usr/local/bin
     - /usr/bin
     - /bin
     - /usr/sbin
     - /sbin

Start by reproducing concurrent gem install runs with prism, ffi, or sqlite3, then trace the RubyGems installation path from the gem install entry point. Compare the two processes' extraction and native-extension compilation steps; done means simultaneous installs no longer cause one process to lose required headers or fail.