Automations: leader election provides no cross-window safety and costs liveness — replace with transactional run claims

## Summary

`AutomationLeaderElection` (`src/vs/sessions/contrib/automations/browser/automationLeaderElection.ts`) exists to guarantee a single automation scheduler across windows. It cannot do that: its claim protocol is built on `IStorageService.store()` / `get()`, which are per-window in-memory cache operations, so the "verify we won by reading it back" step provably always succeeds for the writer. Meanwhile a genuinely atomic compare-and-swap landed four weeks later (`IAutomationStorageService.compareAndSwap`) and already backs the ledger.

Proposal: delete leader election and move the exactly-once guarantee into the CAS transaction, where it can actually be enforced. This cannot be a pure deletion — leader election is currently load-bearing for three other things, listed below.

## Why the current mechanism doesn't work

### 1. The read-back verification is a no-op

```ts
// Write a nonce and verify we won by reading it back (narrows dual-leader window).
const nonce = generateUuid();
const writeOk = this.writeLeader({ instanceId: this._instanceId, heartbeatAt: now, nonce });
...
const verify = this.readLeader();
if (verify?.instanceId === this._instanceId && verify.nonce === nonce) {
```

`store()` → `Storage.set()` writes into the window's own in-memory `cache` synchronously and only schedules a throttled flush (`DEFAULT_FLUSH_DELAY = 100`; `base/parts/storage/common/storage.ts:117,284,292`). `get()` → `Storage.get()` reads that same cache (`storage.ts:219`). So `verify.nonce === nonce` is **always** true for the window that just wrote — the comment is false in every real environment.

Another window's record only becomes visible later, via `onDidChangeItemsExternal` → `acceptExternal` (`storage.ts:145-157`); Electron adds a further ~100 ms debounce in `platform/storage/electron-main/storageIpc.ts`. `AutomationLeaderElection` never subscribes to storage change events — it only polls every 30 s.

### 2. Consequences

- **Dual leader on simultaneous cold start.** Both instances construct, both see the slot claimable, both write, both "verify" their own write, and both dispatch immediately (the scheduler kicks a startup tick from the `isLeader` autorun, `automationScheduler.ts:74-81`). They converge only on a later poll, and hidden web tabs' timers can be throttled well past 30 s.
- **Zero leaders is also reachable.** When `!claimable`, `evaluate()` returns *without writing* (lines 87-93), so an interleaving where each side sees the other's fresh record leaves nobody in charge until a record goes stale.
- **~90–120 s of dead automations after every crash**, including on desktop where there is only one candidate: the crashed record isn't claimable until `now - heartbeatAt > 90_000`, and the replacement only re-evaluates every 30 s.
- **The clean-shutdown tombstone is unreliable.** `releaseIfLeader()` (lines 151-157) calls `store()`, i.e. a 100 ms throttled flush; on abrupt teardown (web tab close in particular) it never lands and the next window eats the full 90 s penalty.

### 3. It isn't the guard for all dispatch paths anyway

"Run now" in the automations list calls the runner directly and never consults the leader — `automationsListWidget.ts:545` passes a hardcoded `leaderWindowId: 0`. A manual run and a scheduler tick can already both pass the `getActiveRunFor` check (`automationRunner.ts:70`) before either commits, within a single window.

### 4. `leaderWindowId` is write-only, and its doc comment is wrong

```ts
/** Window that claimed this run; the leader-election guard uses it to avoid duplicate execution across windows. */
readonly leaderWindowId: number;
```

(`workbench/contrib/chat/common/automations/automation.ts:114-115`). Nothing reads it in production — the only reads are test assertions.

### 5. The tests pass only because of the fixture

`automationLeaderElection.test.ts:32-40` constructs both "windows" over a single `InMemoryStorageService` — one shared synchronous object, precisely the topology that cannot occur in production (each window owns its own `Storage` cache). The `readback ... detects a competing concurrent write` test (106-126) injects the competitor's write synchronously *inside* `store()`; no real backend can do that, so it validates an unreachable branch.

## Why this is now redundant

Timeline:

- **2026-06-30** — leader election lands with the automations foundation (#323745, `041ae8b0c91`). Still the only commit to touch the file.
- **2026-07-25** — `IAutomationStorageService.compareAndSwap` lands (#327110): main-process CAS over the storage IPC channel for Electron (`electron-main/storageIpc.ts:150-161` — synchronous get/compare/set with no await in between), and a single IndexedDB `readwrite` transaction for web (`base/browser/indexedDB.ts:138-172`).
- **2026-07-26** — storage/revision hardening (#327486).

`AutomationService.mutateLedger` (`automationService.ts:358-395`) already retries on CAS failure and re-runs the mutation against fresh state. That is the correct place to enforce "this occurrence runs once" — it holds across windows *and* tabs by construction, with no failover gap.

## What leader election is load-bearing for today

Removal must be paired with replacements for these, or it regresses behaviour:

1. **Duplicate dispatch.** `recordRunStart` (`automationService.ts:271-304`) appends the run unconditionally and advances `lastRunAt`/`nextRunAt`; there is no active-run or occurrence check inside the mutation. The only guard is `getActiveRunFor` (330-332), which reads a local observable.
2. **Crash recovery.** `markStaleRunsFailed` (334-354) fails *every* `pending`/`running` run regardless of owner, on the first tick after gaining leadership (`automationScheduler.ts:125-129`). Safe only under a single-scheduler assumption.
3. **Global serial dispatch.** `_pendingRuns` (`automationScheduler.ts:106-114`) plus the sequential `await` in `dispatchDue` mean automations run one at a time. Per-automation claims alone would let two instances run *different* automations concurrently, losing that limit.
4. **The startup tick.** The immediate crash-recovery + catch-up pass happens only because the election sets `isLeader = true` synchronously in its constructor (line 63) before the autorun is registered. Deleting the election naively defers it to the first 60 s timer tick.

## Proposed change

- [ ] Add a transactional claim inside `mutateLedger`: turn `recordRunStart` into `tryClaimRun`, returning a discriminated result (`claimed` / `alreadyActive` / `occurrenceTaken` / `notFound`) instead of an unconditional `IAutomationRun`. Inside the mutation, reject if the automation already has a `pending`/`running` run, and — for non-manual triggers — if the occurrence being claimed no longer matches. Persist the claimed occurrence (`scheduledFor`) on the run instead of inferring it from `nextRunAt`.
- [ ] Return the authoritative automation snapshot from the claim. The runner currently derives target/prompt/options from its possibly-stale input (`automationRunner.ts:83-112`) before claiming, so a concurrent edit can result in claiming current state but executing an old prompt.
- [ ] Replace the unconditional startup sweep with lease-based expiry: store a full owner UUID (not the 32-bit `stringHash` at `automationScheduler.ts:144`) and an expiry at claim time — the deadline is already known from `chat.automations.runTimeoutMinutes` — and only fail runs whose lease has expired. Note that "sweep only my own window's runs" does **not** work: `instanceId` is a fresh UUID per window construction, so a restarted window would never sweep the crashed owner's runs, and manual runs record `0`.
- [ ] Decide whether global serial dispatch is meant to hold across instances. If yes, enforce the active-run limit inside the same transaction; if no, document that it is per-instance.
- [ ] Add an explicit startup tick to replace the `isLeader` autorun.
- [ ] Delete `automationLeaderElection.ts` and its test, `IAutomationSchedulerCoreOptions.leaderElection`, and `FakeLeaderElection` in `automationScheduler.test.ts`; clean up the orphaned `chat.automations.leader` application-storage key.
- [ ] Fix or fold away the `leaderWindowId` doc comment.
- [ ] Test with two *independent* storage caches over one shared async backend, covering simultaneous cold start, concurrent edit-during-claim, manual + scheduled collision, and post-crash recovery.

## Related defects found while investigating

- `updateRun` (`automationService.ts:306-328`) performs no status-transition validation, so a run marked `failed` by a stale sweep can be flipped straight back to `running` by its owner (`automationRunner.ts:114`).
- The run-timeout path looks up "the active run" rather than retaining its own run id (`automationScheduler.ts:177-186`), so under any duplication it can fail a different run than the one that timed out.

## Caveats checked

- **Desktop is effectively single-window, but not by construction.** `openAgentsWindow` → `ensureAgentsWindow` passes `forceNewWindow: true` (`windowsMainService.ts:326-335`), but `open()` reuses an existing window on the agents workspace first (640-669), so the normal sequential path never creates a second one. There is still a TOCTOU gap: `findWindowOnWorkspaceOrFolder` matches on `openedWorkspace` (`windowsFinder.ts:44-55`), which is `_config?.workspace` (`windowImpl.ts:623`) and isn't populated until `window.load()` — after the async agents-profile creation. Two overlapping opens can both miss each other. Concurrent Agents windows are therefore not a *designed* state, but not impossible either, which argues for a transactional claim rather than for keeping a probabilistic election.
- **Web multi-tab is a real, supported topology.** `vs/sessions/sessions.web.main.internal` is a shipped build entry (`build/buildfile.ts:37`, `build/gulpfile.vscode.web.ts:119`), and IndexedDB application storage is per-origin, so tabs share one ledger and one leader key.
- **The Electron CAS is main-process-scoped**, not a SQLite transaction (`electron-main/storageIpc.ts:150-161`). Two Electron main processes sharing one `state.vscdb` would not be serialized (SQLite storage assumes a single client — `base/parts/storage/node/storage.ts:44`). That is a pre-existing corner case, and leader election is strictly weaker there, since it runs on the same layer with no atomicity at all.