A single transient metadata-server timeout permanently disables credentials for the life of the process

### Environment

- `google-cloud-env` 2.4.0 (also confirmed in 2.2.1; the code is identical on `main`)
- Ruby 3.3, Linux
- Long-lived server processes (GitLab's puma and sidekiq workers) on a GCE VM

### Summary

`ComputeMetadata` caches the outcome of its metadata-server probe for the life of the
process **in both directions**. Once a probe fails past the 60 s warmup window,
`@existence` is pinned to `:no` and never re-probed. Every subsequent Application
Default Credentials lookup in that process then fails with

```text
RuntimeError: Your credentials were not found. To set up Application Default
Credentials for your environment, see
https://cloud.google.com/docs/authentication/external/set-up-adc
```

even though the metadata server was healthy again milliseconds later. The process
never recovers; only a restart clears it.

The defaults make this easy to hit: `DEFAULT_OPEN_TIMEOUT = 0.1`,
`DEFAULT_REQUEST_TIMEOUT = 0.5`. A 100 ms hiccup is enough.

### Why we think this is a bug rather than intended caching

The failure is fed by the transport rescue in `internal_lookup`:

```ruby
rescue *TRANSIENT_EXCEPTIONS
post_update_existence false
raise MetadataServerNotResponding
```

Exceptions the gem itself classifies as **transient** produce a **permanent** verdict.
`post_update_existence` writes `@existence = :no`, and `check_existence` then
short-circuits on it forever:

```ruby
return current if [:no, :confirmed].include? @existence
```

Caching `:confirmed` for the life of the process is sound — a process does not move off
GCP. Caching `:no` is not: it promotes one timed-out request into a permanent statement
about the environment. That asymmetry is the defect.

The `:no` verdict also conflates two very different things: "the SMBIOS gate says we are
definitely not on GCP", which is genuinely permanent, and "one HTTP request timed out",
which is not.

### Impact

Each worker latches independently, so the symptom presents as intermittent rather than
as an outage. In our case a poisoned puma process failed every object-storage operation
it handled while its siblings were fine — users saw HTTP 400 on merge request report
endpoints, and a page refresh appeared to fix it because a healthy process served the
retry. It went undiagnosed for about a week for exactly that reason.

### Reproduction

Standalone, no GCP access needed — a fake metadata server on localhost with
`GCE_METADATA_HOST` pointed at it.

```
gem install google-cloud-env -v 2.4.0
ruby metadata_latch_repro.rb
```

metadata_latch_repro.rb

```ruby
#!/usr/bin/env ruby
# frozen_string_literal: true

# Reproduction: google-cloud-env caches a *negative* metadata-server verdict for
# the life of the process. One probe that times out past the 60 s warmup pins
# @existence = :no permanently, and every later Application Default Credentials
# lookup in that process fails, even though the metadata server is healthy again
# milliseconds later.
#
# No GCP access required. A fake metadata server runs on localhost and
# GCE_METADATA_HOST points at it.
#
# gem install google-cloud-env -v 2.4.0
# ruby metadata_latch_repro.rb
#
# K_SERVICE / K_REVISION / K_CONFIGURATION are set only to get past the SMBIOS
# hardware gate in gce_check, which would otherwise decide "definitely not on
# GCP" before any HTTP happens. On a real GCE VM or Cloud Run revision they are
# unnecessary; they are not part of the defect.
#
# warmup_time is set to 0 so the run takes seconds instead of a minute. It is
# equivalent to any process that has been alive longer than the default 60 s
# warmup, which is every long-lived worker in production.

require "socket"

# A metadata server that answers correctly, or stalls past the client's request
# timeout when @mode is :blip.
class FakeMetadataServer
attr_accessor :mode

def initialize
@mode = :healthy
@server = TCPServer.new "127.0.0.1", 0
@requests = 0
@thread = Thread.new { accept_loop }
end

def address
"127.0.0.1:#{@server.addr[1]}"
end

attr_reader :requests

def stop
@thread.kill
@server.close
end

private

def accept_loop
loop do
socket = @server.accept
Thread.new { handle socket }
end
rescue IOError
nil
end

def handle socket
socket.gets until socket.gets.to_s.strip.empty?
@requests += 1
# A blip: the connection opens, but no response arrives before the client's
# 500 ms request timeout. This is what a momentarily overloaded metadata
# server looks like from the client side.
sleep 1.0 if @mode == :blip
socket.print "HTTP/1.1 200 OK\r\nMetadata-Flavor: Google\r\nContent-Length: 2\r\n\r\nok"
rescue StandardError
nil
ensure
socket.close rescue nil
end
end

server = FakeMetadataServer.new
ENV["GCE_METADATA_HOST"] = server.address
ENV["K_SERVICE"] = ENV["K_REVISION"] = ENV["K_CONFIGURATION"] = "repro"

require "google/cloud/env"
require "google/cloud/env/version"

def fresh_env
env = Google::Cloud::Env.new
env.compute_metadata.warmup_time = 0
env
end

def report label, value
puts format(" %-46s %s", label, value)
end

puts "google-cloud-env #{Google::Cloud::Env::VERSION}, metadata host #{ENV['GCE_METADATA_HOST']}"
puts

# Control. Proves the HTTP probe is what decides existence here, so the runs
# below are not being answered by the SMBIOS gate or by a cached verdict.
puts "[control] healthy server"
env = fresh_env
report "metadata?", env.metadata?
report "existence", env.compute_metadata.existence_immediate
puts

# The defect. A single blip while the server is briefly unresponsive, then the
# server is healthy for every subsequent call — and the process never recovers.
puts "[defect] one blip, then a healthy server forever"
env = fresh_env
server.mode = :blip
report "metadata? during the blip", env.metadata?
report "existence", env.compute_metadata.existence_immediate
server.mode = :healthy
before = server.requests
3.times { env.metadata? }
report "metadata? after the blip (server healthy)", env.metadata?
report "existence", env.compute_metadata.existence_immediate
report "HTTP requests made by those 3 calls", server.requests - before
puts

# The warmup boundary. The identical blip, on a process still inside its warmup
# window, leaves existence :unconfirmed and recovers on the next call. Only the
# warmup comparison in post_update_existence separates recovery from permanent
# failure, so what breaks a worker is being alive for more than 60 seconds.
puts "[control] the same blip, inside the warmup window"
env = Google::Cloud::Env.new
server.mode = :blip
report "metadata? during the blip", env.metadata?
report "existence", env.compute_metadata.existence_immediate
server.mode = :healthy
before = server.requests
report "metadata? after the blip", env.metadata?
report "existence", env.compute_metadata.existence_immediate
report "HTTP requests made by that call", server.requests - before
puts

# The asymmetry. A confirmed verdict cached forever is sound; the process is on
# GCP and will stay there. A negative verdict cached forever is not: it encodes
# a 100 ms network timeout as a permanent fact about the environment.
puts "[asymmetry] confirmed first, then the same blip"
env = fresh_env
env.metadata?
report "existence after a healthy probe", env.compute_metadata.existence_immediate
server.mode = :blip
env.metadata?
server.mode = :healthy
report "existence after the blip", env.compute_metadata.existence_immediate
puts

server.stop

puts <<~SUMMARY
Expected: the [defect] run recovers once the server answers again.
Actual: existence is :no, check_existence short-circuits on it, and the
three later calls make zero HTTP requests. In a real process every
Application Default Credentials lookup from that point on raises
"Your credentials were not found", until the process restarts.
SUMMARY
```

Output:

```text
google-cloud-env 2.4.0, metadata host 127.0.0.1:39961

[control] healthy server
metadata? true
existence confirmed

[defect] one blip, then a healthy server forever
metadata? during the blip false
existence no
metadata? after the blip (server healthy) false
existence no
HTTP requests made by those 3 calls 0

[control] the same blip, inside the warmup window
metadata? during the blip false
existence unconfirmed
metadata? after the blip true
existence confirmed
HTTP requests made by that call 1

[asymmetry] confirmed first, then the same blip
existence after a healthy probe confirmed
existence after the blip confirmed
```

The two controls are what make the middle run meaningful:

- The first proves the HTTP probe, not the SMBIOS gate, is deciding existence here.
- The second runs the **identical** blip inside the warmup window: existence stays
`:unconfirmed` and the next call recovers, having actually made a request. Only the
warmup comparison in `post_update_existence` separates recovery from permanent
failure — so what breaks a worker is simply having been alive for more than 60 s.

`HTTP requests made by those 3 calls: 0` is the short-circuit: after latching, the
library stops talking to the metadata server entirely.

### Suggested fix

We have no strong preference between these, and would rather defer to you:

1. Do not latch `:no` from a transport failure at all. Reserve `:no` for conclusive
negatives (the SMBIOS/environment gate, or a server that answers with the wrong
flavor), and leave a timed-out probe at `:unconfirmed` so the next lookup retries.
2. Give the `:no` verdict an expiry, so `check_existence` re-probes after some interval
rather than short-circuiting forever.

Either would have prevented this. (1) seems closer to the existing intent, given that
the failing path is already named `TRANSIENT_EXCEPTIONS`.

### Workaround, for anyone who finds this first

We install a Rails initializer that calls `reset_existence!` and then
`ensure_existence(timeout: 10.0)` at boot. The reset matters: `Env.new` runs at file
scope during `Bundler.require`, so the warmup clock is already stamped by the time
initializers run. Since `:confirmed` is cached just as permanently, confirming once at
boot makes the `:no` branch unreachable for the life of the process. That is a
workaround for the asymmetry, not a fix for it.

Start by tracing the named internal_lookup, post_update_existence, and check_existence paths, including TRANSIENT_EXCEPTIONS, then run the metadata_latch_repro.rb reproduction. Done means a transient metadata-server failure does not permanently prevent a later healthy probe from recovering, while the existing confirmed behavior remains intact.

Written by the indexing model from the issue text.