Edge Function served code that was never deployed for ~75 min after first deploy (version 1, one deployment id, two regions)

This incident is over, and we are not asking for anything to be restored. The cause is
open, and it applies to every future deployment — which is why we are reporting it.

Function: score-compute · Project region: EU (project ref withheld) · Date: 2026-08-25

Summary

For roughly 75 minutes after the first ever deployment of an Edge Function,
invocations were served by code that does not match the deployed source. Both regions we
observed were affected, and they caught up at different times: eu-central-1 some time
between 11:51 and 13:01 UTC, eu-west-1 between 12:31 and 13:24 UTC. During the overlap,
identical stored inputs produced different persisted results depending on which region served
the call. It ended on its own, without any action on our side.

Facts

The two observable differences

1. A key that the source sets unconditionally was missing. The handler writes a JSON
   object that always contains capCutoff (a hardcoded string constant in a shared module,
   no branch, no environment lookup). No row written before 13:01 UTC contains it. Every
   row written after does.
2. A different numeric result. Unchanged stored inputs produced two different results:
   B in eu-west-1 (12:31:12) and A in eu-central-1 (11:51:43, 13:01:06,
   13:02:46 x3). Result B can only be produced by a code path that exists in an earlier
   revision of a shared module — a revision that was never deployed as part of this
   function, because this function has only ever had version 1. The concrete numbers are
   user data and are left out; what matters here is that one of the two is unreachable from
   the deployed source.
   Scope of this claim: it rests on the invocations listed in the timeline, each of which is
   the only invocation between the write and our read. We are not claiming a region split
   for the full window — eu-central-1 served a build without capCutoff at 11:51 as well.

Timeline (all UTC, same account, same stored inputs)

11:44:35  deploy (version 1, first ever)
11:51:43  eu-central-1  -> result A, capCutoff MISSING
12:31:12  eu-west-1     -> result B, capCutoff MISSING
13:01:06  eu-central-1  -> result A, capCutoff present
13:02:46  eu-central-1  -> result A, capCutoff present  (x3, identical)
13:24:46  eu-west-1     -> result A, capCutoff present  <- divergence gone

Every line above is a single invocation whose persisted row can be attributed to it
unambiguously: no other invocation of this function overlapped or fell between it and the
read. Two further eu-west-1 invocations at 12:23 also produced a stale result, but the row
we read cannot be tied to a specific invocation, so they are left out of this report.

Invocation paths: eu-central-1 via HTTPS from our own host; eu-west-1 via net.http_post
from the database with x-region: eu-west-1. Both hit the same function URL.

Region and timing are taken from your own function_edge_logs (x_sb_edge_region,
execution_time_ms, deployment_id), not only from the response headers we recorded — the
two sources agree on every invocation.

One deployment id throughout

All 45 invocations of this function on 2026-08-25 between 11:40 and 14:00 UTC report

deployment_id = <project-ref>_11c327e5-a673-4507-a617-7a0a779a612e_1
version       = 1

Only the project-ref prefix is redacted here; the string is byte-identical in all 45 rows.
The divergent and the correct results carry the same deployment id. Whatever served the
stale code was not recorded as a different deployment on your side either.

What we ruled out on our side (so you don't have to ask)

1. A stored value or configuration. The capCutoff string is a compile-time constant in
   the source. No table, no config row, no environment variable. Database-side filtering was
   checked: the JSON key has been explicitly allowed since an earlier migration, unchanged.
2. A second function bundling an older copy of the shared module. Exactly two functions
   import it. The other one returns HTTP 410 unconditionally and writes to a different table;
   the calling trigger targets score-compute by fixed URL, and every response we recorded
   is the score-compute payload.
3. Module-level state cached per instance. The only module-level values are the project
   URL, the service key, and two secrets — none influences the computation. The database
   client is constructed per request inside the handler, so there is no per-instance
   schema cache in our code.
4. Input data changing between calls. The inputs are stored rows; the divergent results
   bracket calls with identical stored inputs, and the post-13:01 result is stable across
   five subsequent invocations.

What we would like to understand

• Where does a first deployment (version 1, never redeployed) obtain a module revision
  that was never deployed as part of it — and why is it served under that deployment's own
  id?
• Can two regions serve different code for the same deployment id at the same moment, and
  for how long? Our two observations differ by roughly 25 minutes in when they caught up.
• Is there a way to verify propagation before we consider a deployment complete? We can
  currently only detect this by comparing results across regions after the fact.

Impact

For roughly 75 minutes, two revisions of the same formula were live at once, and which one
a call got depended on the region that served it. The results are user-visible and were
persisted, carrying a version identifier that named the deployed revision. A later scheduled
run overwrote them, so nothing incorrect remains. What concerns us is not the cleanup: while
the window was open, neither your records nor ours distinguished the two — the divergent and
the correct results are logged under one deployment id.

Everything above is taken from the platform's own records (function_edge_logs,
deployment_id, version) and from rows the function itself wrote — none of it depends on
instrumentation of ours. Happy to supply anything further that helps reproduce this.