Throttling risk: trigger fan-out and shared schedules can exceed Cognito Identity Pool GetOpenIdTokenForDeveloperIdentity quota (50 RPS)

## Summary

**Priority: Low.** At larger user counts, the event-trigger code paths can issue a burst of `GetOpenIdTokenForDeveloperIdentity` calls that exceed the Cognito Identity Pool default quota (**50 RPS**), causing throttling and dropped trigger executions. This issue proposes two small, self-contained mitigations.

This is a scalability hardening item only. It does not affect correctness at small scale and is not a security issue.

## Background

Triggers obtain per-user credentials via Cognito Identity Pool **Developer Authenticated Identities**:

- The Trigger Lambda calls `GetOpenIdTokenForDeveloperIdentity` (default quota **50 RPS**, adjustable).
- The downstream agent runtime then calls `GetCredentialsForIdentity` (default quota 200 RPS).

The tightest limit is `GetOpenIdTokenForDeveloperIdentity` at **50 RPS**. Two trigger patterns can concentrate calls and exceed it:

### 1. Custom-event fan-out (most concentrated)

`packages/trigger/src/handlers/custom-event-handler.ts` queries all triggers subscribed to an `eventSourceId` (GSI2) and invokes them **fully in parallel** with `Promise.allSettled`:

```ts
const results = await Promise.allSettled(
triggers.map((trigger) =>
invokeTrigger(trigger, event, authService, agentInvoker, executionRecorder)
)
);
```

Each `invokeTrigger` performs one `GetOpenIdTokenForDeveloperIdentity` call. A single event with N subscribers fires N near-simultaneous calls inside one Lambda invocation. This cannot be mitigated by time-spreading because all subscribers fire on the same event.

### 2. Shared schedule times

`packages/backend/src/services/scheduler-service.ts` creates each EventBridge Schedule with `FlexibleTimeWindow: { Mode: 'OFF' }`:

```ts
FlexibleTimeWindow: {
Mode: 'OFF',
},
```

With `Mode: 'OFF'`, all schedules set to the same cron (e.g. `cron(0 9 * * ? *)`, or users naturally clustering on `HH:00`) dispatch at the same instant, producing a synchronized burst.

## Peak RPS estimates (assume 5,000 users, 1 trigger/user, quota = 50 RPS)

| Scenario | Assumption | Peak RPS | vs 50 RPS |
|---|---|---:|---:|
| All same cron `9:00` (1s dispatch window) | synchronized | ~5,000 | x100 |
| All same cron `9:00` (5s dispatch window) | synchronized | ~1,000 | x20 |
| Event fan-out, 500 subscribers, <1s | single Lambda parallel | ~500 | x10 |
| Hourly `:00` clustering (167 users at same second, 3s) | natural clustering | ~56 | x1.1 |
| 24h uniform spread (per-second jitter) | spread | ~0.06 | ok |

Even modest natural clustering (~167 users at the same second) already exceeds the quota, and event fan-out exceeds it with as few as ~50 subscribers.

## Proposed fixes

### Fix 1 — Bound fan-out concurrency in `custom-event-handler.ts`

Replace the unbounded `Promise.allSettled(map(...))` with a concurrency-limited runner so that at most `MAX_CONCURRENCY` (e.g. 20) credential exchanges are in flight at once. Add a small dependency such as [`p-limit`](https://www.npmjs.com/package/p-limit) (or a tiny inline limiter to avoid the dependency).

```ts
import pLimit from 'p-limit';

// Keep below the GetOpenIdTokenForDeveloperIdentity quota (default 50 RPS),
// leaving headroom for concurrent Lambda invocations.
const FANOUT_CONCURRENCY = Number(process.env.TRIGGER_FANOUT_CONCURRENCY ?? 20);

const limit = pLimit(FANOUT_CONCURRENCY);
const results = await Promise.allSettled(
triggers.map((trigger) =>
limit(() =>
invokeTrigger(trigger, event, authService, agentInvoker, executionRecorder)
)
)
);
```

Inline alternative (no new dependency):

```ts
async function mapWithConcurrency(
items: T[],
concurrency: number,
fn: (item: T) => Promise
): Promise[]> {
const results: PromiseSettledResult[] = new Array(items.length);
let next = 0;
const workers = Array.from({ length: Math.min(concurrency, items.length) }, async () => {
while (true) {
const i = next++;
if (i >= items.length) return;
try {
results[i] = { status: 'fulfilled', value: await fn(items[i]) };
} catch (reason) {
results[i] = { status: 'rejected', reason };
}
}
});
await Promise.all(workers);
return results;
}

// usage
const results = await mapWithConcurrency(triggers, FANOUT_CONCURRENCY, (trigger) =>
invokeTrigger(trigger, event, authService, agentInvoker, executionRecorder)
);
```

### Fix 2 — Enable a flexible time window for schedules in `scheduler-service.ts`

Use EventBridge Scheduler's built-in `FlexibleTimeWindow` to spread synchronized cron schedules across a window automatically (no custom jitter logic needed):

```ts
const FLEX_WINDOW_MINUTES = Number(process.env.SCHEDULE_FLEX_WINDOW_MINUTES ?? 5);

// in CreateScheduleCommand (and the UpdateSchedule path around line 357)
FlexibleTimeWindow: FLEX_WINDOW_MINUTES > 0
? { Mode: 'FLEXIBLE', MaximumWindowInMinutes: FLEX_WINDOW_MINUTES }
: { Mode: 'OFF' },
```

A 5-minute window spreads otherwise-synchronized schedules across 300 seconds. Example: 5,000 users all on the same cron drop from ~5,000 RPS (1s) to ~17 RPS average over the window. This should be documented as a best-effort timing (executions may run up to N minutes after the nominal time).

### Fix 3 (optional) — Quota increase + retry

- Request a Service Quotas increase for `GetOpenIdTokenForDeveloperIdentity` if high trigger volume is expected.
- Add exponential backoff / retry for `ThrottlingException` / `TooManyRequestsException` on the Identity calls in the trigger auth path as defense-in-depth.

## Acceptance criteria

- [ ] Custom-event fan-out limits concurrent credential exchanges to a configurable bound (default ~20).
- [ ] Schedule creation/update supports a configurable `FlexibleTimeWindow` (default FLEXIBLE, e.g. 5 min) to de-synchronize shared cron times.
- [ ] Behavior is configurable via env vars and defaults are safe for the documented "few hundred users" PoC scale.
- [ ] Unit tests cover the concurrency limiter and the schedule flexible-window option.

## Notes

- The steady-state RPS is negligible thanks to in-memory credential caching; this issue only concerns **burst** patterns.
- No account-specific or confidential information is involved; this describes application behavior only.

Start by reading packages/trigger/src/handlers/custom-event-handler.ts and packages/backend/src/services/scheduler-service.ts, including the schedule update path around line 357. Run the existing test suite and locate the relevant unit-test patterns before implementing configurable fan-out concurrency and FlexibleTimeWindow settings. Done means both options have safe defaults, honor their environment variables, and have unit tests covering concurrency limiting and schedule-window configuration.

Written by the indexing model from the issue text.