Dedup sidecar is committed before batched trace export is flushed, which can suppress retries after export failure

Summary

Completed Codex turn IDs are currently written to the dedup sidecar before the plugin has successfully flushed its batched spans to Langfuse.

This creates a data-loss path:

1. convertRollout() emits the spans for a completed turn.
2. The turn ID is immediately appended to <rollout>.langfuse.
3. runHook() later calls instrumentation.shutdown().
4. shutdown() performs spanProcessor.forceFlush().
5. If the batch export fails or times out, the sidecar already records the turn as uploaded.
6. A later Stop hook loads that sidecar and skips the turn, so the failed export is not retried.

A transient export failure can therefore become a permanent missing trace.

Current ordering

In plugins/tracing/src/trace.ts, a completed turn is marked immediately after the observation tree is created:

if (turn.completed && turn.turnId) {
  uploaded.add(turn.turnId);
  await markTurnUploaded(rolloutFile, turn.turnId);
}

However, the plugin uses batched exporting in plugins/tracing/src/instrumentation.ts:

exportMode: "batched",

and the actual awaited flush happens later:

shutdown: async () => {
  await spanProcessor.forceFlush();
  await provider.shutdown();
  setLangfuseTracerProvider(null);
}

runHook() calls convertRollout() first and instrumentation.shutdown() afterward.

This means the persistent dedup receipt is committed before the awaited delivery boundary.

Expected behavior

A completed turn should only be recorded in the dedup sidecar after its batched spans have successfully passed the awaited flush/export boundary.

If the export fails, the sidecar should remain uncommitted for that turn so the next Stop hook can retry it.

This also matches the existing failure trade-off in markTurnUploaded(): failure to persist the sidecar may cause a duplicate later, which is safer than silently losing the trace.

Suggested direction

One possible approach:

1. Let convertRollout() return the completed top-level turn IDs that were emitted, without persisting them yet.
2. Explicitly forceFlush() the instrumentation.
3. Only after a successful flush, append those turn IDs to the sidecar.
4. Then shut down the provider.

This keeps conversion, delivery confirmation, and dedup receipt commitment in the correct order.

Suggested regression test

This can be tested without real Langfuse credentials or network access because the existing instrumentation tests already mock LangfuseSpanProcessor.

A regression test could:

1. Process a completed fixture rollout.
2. Make the mocked forceFlush() fail.
3. Assert that the turn ID is not written to the .langfuse sidecar.
4. Run the same rollout again with a successful flush.
5. Assert that the turn is emitted again and the sidecar is then committed.
6. Run it a third time.
7. Assert that the committed sidecar now prevents another export.

It would also be useful to cover the default fail_on_error=false path, since the hook intentionally fails open but should not convert an export failure into a permanent dedup receipt.

Impact

This matters most for short-lived Stop-hook execution, where the final forceFlush() is the point at which buffered spans are actually handed off before process exit.

The current ordering can make temporary Langfuse/network/export failures unrecoverable even though later Stop hooks would otherwise provide a natural retry opportunity.