Prevent concurrent direct-S3 mutations from wedging Witan's Omnigraph store

Expected Behavior

Concurrent Witan mutations against a direct S3 target should either be serialized or return a bounded, typed concurrency/recovery result. Omnigraph recovery should converge without permanently blocking reads, writes, and repair diagnostics.

Current Behavior

With Witan 0.36.0, witan-core 0.37.0, and Omnigraph 0.11.0, two concurrent mutations against one format-9 S3 graph left an OCC recovery sidecar that permanently quarantined the graph:

OCC recovery sidecar '01M2V544QY00SSGZA0PSRK3B9X'
found original commit id '01M2V54457QR0XB7ZW47H5V88H'
but its manifest delta differs

Every subsequent query and mutation failed with the same error. omnigraph repair also refused to open the graph while the sidecar remained active.

Steps to Reproduce

1. Configure Witan to use a direct s3://<bucket>/graph-v9.omni target with Omnigraph 0.11 storage format 9.
2. Create an active workflow project.
3. Submit workflow_session_start and task_create concurrently against that target, for example in one Promise.all or parallel MCP tool batch.
4. Observe one call report success and the other fail during OCC recovery.
5. Run any Witan read or write, or omnigraph repair --store s3://<bucket>/graph-v9.omni --json.
6. Observe the permanent sidecar/manifest-delta failure above.

Possible Solution

• Serialize mutations within witan serve, keyed by resolved target URI, even where a local cross-process flock is impossible. Continue documenting that separate processes writing directly to S3 remain uncoordinated.
• Upstream in Omnigraph, make this recovery state converge deterministically or provide a supported command that can inspect and resolve it without manually moving the active sidecar.
• Add a concurrent-writer regression test against an S3-compatible backend. Assert that the losing writer receives a typed conflict/recovery response and that later reads and repairs succeed.

Additional Details

Witan deliberately skips its filesystem lock for S3 targets because they are not locally lockable: https://github.com/mitodl/agent-kit/blob/main/packages/witan-core/witan_core/omnigraph.py#L1138-L1148

The recovery sidecar described four affected tables: BelongsTo, ForProject, CodeBranch, and WorkflowSession. It recorded expected version 2, post-commit pin 3, rollback outcomes from version 3 to 2, and effect_phase: Armed.

Inspection after preserving and moving the sidecar showed:

• The original graph commit was present and its entity changes were exactly the intended session node, code-branch node, and two edges.
• The rollback commit ID recorded in the sidecar was absent from commit history.
• omnigraph repair classified only those four datasets as suspicious, each with published version 3 and Lance HEAD version 4 (Restore). All other datasets reported no_drift.
• A verified backup followed by omnigraph repair --confirm --force realigned the four tables. A subsequent dry run reported no_drift for all 27 datasets, zero active recovery sidecars, and successful reads and writes.

Start in packages/witan-core/witan_core/omnigraph.py around lines 1138-1148, where filesystem locking is skipped for S3 targets, and reproduce the concurrent workflow_session_start/task_create calls against the format-9 S3 graph. Trace the OCC recovery sidecar and omnigraph repair behavior; done means concurrent mutations produce a bounded typed outcome, recovery converges, and later reads, writes, and repair diagnostics succeed.