Improve operator UX around PROM_REMOTE_WRITE_TOKEN (make remote_write opt-in, document the token)

Context: ObolNetwork/obol-infrastructure#3008 — a number of partner nodes are pushing metrics to vm.monitoring.gcp.obol.tech/write with credentials vmauth rejects. Per @OisinKyne, for most of them it's not that we invalidated a token — they never set one at all.

That's a launcher UX problem, and it's fixable here.

Why it happens

prometheus/prometheus.yml.example declares remote_write unconditionally, and prometheus/run.sh hard-exits when PROM_REMOTE_WRITE_TOKEN is empty. So today:

• No token → Prometheus crash-loops, and the operator loses their local Grafana too, for a feature they may never have wanted.
• Junk token (sample placeholder, Bearer <tok> pasted whole, trailing newline) → renders fine, ships 401s at vmauth indefinitely.
• Either way the only feedback is a line in docker compose logs prometheus, which nobody reads.

And nothing in the repo says where a token comes from. .env.sample.mainnet:201 reads "Prometheus remote write token used for accessing external prometheus" — it doesn't say it's Obol's, doesn't say it's optional, doesn't link anywhere. Zero hits for PROM_REMOTE_WRITE_TOKEN in any .md.

Proposal

A. Make token presence the switch

Move the remote_write block out of prometheus.yml.example into prometheus/remote-write.yml.example, and have run.sh append it only when the token is set. Drop the exit 1.

if [ -n "$PROM_REMOTE_WRITE_TOKEN" ]; then
  sed -e "s|\$PROM_REMOTE_WRITE_TOKEN|${PROM_REMOTE_WRITE_TOKEN}|g" \
      /etc/prometheus/remote-write.yml.example >> /etc/prometheus/prometheus.yml
else
  echo "WARN: PROM_REMOTE_WRITE_TOKEN unset - local monitoring only." >&2
  echo "WARN: Obol-hosted dashboards and alerts are disabled. Request a token: <link>" >&2
fi

run.sh is our {{ if }} — the conditional layer already exists, we just aren't using it. The local stack keeps working either way (the write_relabel_configs only forward charon/mev-boost metrics, so nothing local is lost), and operators who never onboarded stop reaching vmauth at all.

E. Fix the .env.sample comment

Say what the token is, that it's optional, what you give up without it, and where to get one. Both .env.sample.mainnet and .env.sample.hoodi.

A and E together should remove most of the rejected population, since that population mostly consists of people who never had a credential in the first place.

Not proposing now, but on the table

• B — format validation. A case in run.sh rejecting the sample placeholder, a Bearer prefix, whitespace, wrong length. Catches the paste-the-whole-header class.
• C — verify at startup. One curl -o /dev/null -w '%{http_code}' -X POST against the write endpoint before exec'ing Prometheus; on 401, print the actual diagnosis and the re-request link. Anything else (a 400 on an empty protobuf body means auth succeeded) proceeds. Warn-and-continue, never fatal — a monitoring outage must not stop a validator. This is what turns a silent server-side mystery into a message on the operator's own terminal.
• D — surface it post-start. prometheus_remote_storage_samples_failed_total is already scraped; one alert rule and one local-Grafana panel ("Obol remote-write: OK / 401"). The only option here that catches rotation breakage months after setup, which C cannot.
• F — server-side, tracked in obol-infrastructure#3008. Per-tenant vmauth logging and tokens carrying operator identity, so a 401 is attributable without correlating source IPs against onboarding records. Needed for diagnosis; doesn't improve operator UX on its own.
• G — remove the shared secret. Charon already signs monitoring payloads with the ENR key (charon alpha test --publish does exactly this). Auth by ENR / cluster-lock signature means nothing to distribute and nothing to rotate, and unregistered nodes become impossible by construction. The real fix; worth scoping once F gives us a baseline to measure against.

Mirroring

lido-charon-distributed-validator-node carries the same prometheus/run.sh, and helm-charts/charts/dv-pod has its own path to the same endpoint. Whatever lands here needs to go to both, or we fix a third of the operators.

Start with prometheus/run.sh, prometheus/prometheus.yml.example, and .env.sample.mainnet and .env.sample.hoodi; compare the mirrored prometheus/run.sh and helm-charts/charts/dv-pod paths. Make remote_write conditional on PROM_REMOTE_WRITE_TOKEN, clarify the optional token comments, and verify that local monitoring starts without a token while configured remote write still works.