Reduce Stemcell Distribution Egress Costs via Cloudflare CDN

Summary

Stemcell downloads from https://storage.googleapis.com/bosh-aws-light-stemcells/...
(and equivalent GCS buckets for other IaaS) generate significant GCP egress costs (~2K $ per month).
We can drastically reduce these costs by fronting the GCS buckets with Cloudflare CDN under a community-owned domain such as stemcells.cloudfoundry.org.

Related migration: cloudfoundry/community#926

Motivation

• Stemcells are immutable, versioned, large artifacts — ideal CDN cache candidates.
• Current GCS egress is billed at ~$0.12/GB; Cloudflare egress to end users is free.
• A single cached version is fetched from origin once, then served from edge globally.
• Improves download speed and reliability for the community.

Proposed Architecture

User → https://stemcells.cloudfoundry.org/<iaas>/<version>/<file>.tgz
         │
         └── Cloudflare CDN (cached at edge)
                │  (cache miss only)
                └── storage.googleapis.com/bosh-<iaas>-light-stemcells/...

Example URL mapping:

• Before: https://storage.googleapis.com/bosh-aws-light-stemcells/1.425/light-bosh-stemcell-1.425-aws-xen-hvm-ubuntu-noble.tgz
• After: https://stemcells.cloudfoundry.org/aws/1.425/light-bosh-stemcell-1.425-aws-xen-hvm-ubuntu-noble.tgz

Action Items

Phase 1 — Infrastructure setup

• Create Cloudflare zone (or subdomain delegation) for stemcells.cloudfoundry.org
• Verify stemcell file sizes vs Cloudflare cache limits (Free 512MB / Pro 1GB / Business 5GB / Enterprise unlimited with Cache Reserve)
• Decide Cloudflare plan (or apply for Cloudflare's open-source / OSS sponsorship program)
• Ensure GCS buckets (bosh-aws-light-stemcells, bosh-gce-light-stemcells, bosh-azure-light-stemcells, etc.) have allUsers:objectViewer

Phase 2 — Cloudflare configuration

• Add CNAME stemcells.cloudfoundry.org → storage.googleapis.com (proxied)
• Add Origin Rule: override Host header to storage.googleapis.com
• Add URL Rewrite rule mapping /<iaas>/<path> → /bosh-<iaas>-light-stemcells/<path>
• Configure Cache Rule: Edge TTL = 1 month, Browser TTL = 1 day, cache eligible
• Enable Tiered Cache for better hit ratio
• (Optional) Enable Cache Reserve for large objects

Phase 3 — Release automation integration

• Update stemcell publishing pipeline (bosh-io / stemcells-index and equivalents) to:
  • Continue uploading to GCS as origin of truth
  • Issue a curl -I warm-up request after upload to pre-fill Cloudflare cache
  • Publish CDN URL (stemcells.cloudfoundry.org/...) in release metadata / bosh.io index
• Update bosh.io UI/API to point downloads at CDN URL
• Keep GCS URLs functional for backward compatibility (no breaking change)

Phase 4 — Monitoring & rollout

• Add Cloudflare Analytics dashboard (cache hit ratio, bandwidth saved)
• Compare GCP egress bill before/after (1 month baseline)
• Document new URL pattern in BOSH / stemcell docs
• Announce on cloudfoundry-dev mailing list & Slack #bosh

Phase 5 — Optional follow-up

• Evaluate full migration of origin to Cloudflare R2 (zero egress, no large-file cache limits) if cost savings justify operational change

Success Criteria

• >90% cache hit ratio after warm-up
• Measurable reduction in GCP egress costs (target: >70%)
• No regressions in stemcell download reliability
• Zero breakage for existing consumers using old GCS URLs

Start by reviewing the related migration in cloudfoundry/community#926 and the release automation references to bosh-io/stemcells-index and the bosh.io UI/API. Define the infrastructure, URL mapping, rollout, and backward-compatibility work across the listed phases; done means the success criteria are met without breaking existing GCS URLs.