Sandboxing → Configurable Auto-Resilience for PyFluent

# Sandboxing → Configurable Auto-Resilience for PyFluent

*Expansion of the "Sandboxing" enabler under Agentic AI Platform → Make PyFluent Architecture Agentic-Ready*

## Motivation

The original issue frames sandboxing narrowly, as a way to let users trial changes before committing them. This proposal reframes it more broadly: **encapsulate resilience within PyFluent itself**, so PyFluent can mitigate — never fully eliminate — poor resilience on the Fluent side. This is deliberately decoupled from agentic AI: any user, agentic or not, benefits from a PyFluent that can detect and recover from a broken Fluent session without losing their work. Agentic workflows just make the need more acute, since an autonomous loop has no human in place to notice a hang or a corrupted state and intervene manually.

The result should be a set of **user-configurable resilience modes** — an opt-in capability, not a default behavior change.

## Dependency note

The parent issue lists undo/redo (`get_state()`/`set_state()` snapshotting) as an exploratory item, not yet built. Several modes below lean on "the undo-redo stack" as a lighter alternative to client-side journaling. That dependency should be called out explicitly: until undo/redo lands, journal-replay is the only available recovery mechanism for modes 1 and 2, and the undo-redo-based variant is aspirational rather than a fallback we can ship today.

## Design axes

Rather than four independent, freely-chosen modes, this is better modeled as two orthogonal axes:

1. **Session redundancy** — one live Fluent session, or a warm standby second session.
2. **Recovery source** — replay a command journal (or undo/redo stack), or restore from a case+data checkpoint.

The right choice on axis 2 is not purely a user preference — it depends on **client cardinality** (see below), so PyFluent should be able to select or constrain it rather than exposing it as an unconstrained free choice.

### Modes

| Mode | Sessions | Recovery source | Recovery speed | Steady-state cost | Fidelity |
|---|---|---|---|---|---|
| 1 | Primary + warm secondary | Journal / undo-redo replay | Fast (secondary already running) | 2x compute, 2x license | Good for settings mutations; not guaranteed for solve/mesh ops |
| 2 | Single | Journal / undo-redo replay | Slower (cold start + replay) | 1x compute, 1x license | Same fidelity caveat as mode 1 |
| 3 | Primary + warm secondary | Case+data | Fast | 2x compute, 2x license, high I/O if checkpointed per-command | Faithful, server-truth-based, multi-client safe |
| 4 | Single | Case+data | Slower | 1x compute, 1x license, high I/O if checkpointed per-command | Same as mode 3 |
| **5 (proposed)** | Either | **Checkpoint + journal since last checkpoint** | Fast, bounded | Tunable via checkpoint interval | Bounded fidelity gap, bounded I/O cost |

## Fidelity limits of journal/undo-redo replay

Journal (or undo/redo) replay assumes that re-issuing an identical command sequence into a fresh session reproduces the same end state. That holds for pure settings/state mutations. It does **not** reliably hold for solve or meshing operations: parallel partitioning, thread scheduling, and iterative convergence paths are not guaranteed bit-reproducible across separate runs, even given identical inputs and command order. A "recovered" secondary built purely from journal replay may therefore diverge from what the primary actually held at the moment of failure, specifically for long-running compute steps.

Practical implication: the design should distinguish between two command classes —
- **State/settings mutations** — safely replayable via journal or undo-redo.
- **Long-running compute (solve, mesh)** — not safely replayable; case+data is the only faithful recovery path across these.

This is the underlying reason modes 1/2 are fast-but-not-always-faithful and modes 3/4 are faithful-but-slow: they aren't really substitutable options, they cover different failure surfaces.

## Multi-client correctness

Journal/undo-redo-based recovery assumes PyFluent's client is the sole source of mutations. If another client mutates the same Fluent session outside PyFluent's visibility — the Scheme console, the TUI, or another PyFluent instance — journal replay silently reconstructs a session that never matches what the primary actually held. There's no error signal; the divergence is invisible.

Case+data sidesteps this entirely, since it reads server-side truth rather than replaying client-observed history.

**Recommendation:** PyFluent should detect or be told the client cardinality of a session, and:
- Single-client sessions may use journal/undo-redo replay (modes 1/2) as a lighter-weight option.
- Multi-client sessions should be restricted to case+data-based recovery (modes 3/4/5), since journal fidelity cannot be guaranteed.

## Mode 5: checkpoint + journal hybrid (proposed)

The "glacial" cost attributed to case+data-based recovery is a function of checkpoint *frequency*, not an inherent property of the approach. Instead of saving case+data on every command:

- Take a case+data checkpoint periodically — every *N* commands, every *T* minutes, or after specific high-value operations (post-mesh, post-solve-convergence).
- Maintain a journal (or undo-redo log) only for commands since the last checkpoint.
- On critical failure: restore the last case+data checkpoint, then replay the short journal since that checkpoint.

This bounds both the I/O cost of frequent case+data saves and the replay-time cost of a long journal on a long-running session, and caps the multi-client blind-spot window to the interval between checkpoints rather than the whole session lifetime. This should be the default recommended mode; 1–4 remain useful as the degenerate endpoints (checkpoint interval → ∞ gives pure journal; checkpoint interval → 1 command gives pure case+data).

## Detecting critical failures

Detection is inherently heuristic ("best effort," per the original issue) and should combine multiple signals rather than relying on any one:

1. **Exception introspection** — requires the Fluent server to emit strongly-typed, actionable exceptions; PyFluent extracts as much structured signal as possible.
2. **Transcript streaming** — heuristic pattern matching on Fluent's transcript output. Useful but brittle: patterns are liable to break across Fluent versions or localization changes, so this should be treated as a lower-confidence signal, not a primary one.
3. **OpenTelemetry stream** — potentially higher-fidelity than transcript parsing; worth prioritizing over transcript heuristics where available.
4. **Process monitoring** — detect unexpected termination of Fluent processes.
5. **gRPC health checks** — likely already performed before returning control. Note: a health check against the gRPC control-plane thread may report "alive" even when Fluent is hung in a compute loop unrelated to that thread. Health checks should be treated as necessary but not sufficient for liveness.
6. **Round-trip sanity checks** — e.g. `set_state(val)` followed by `get_state() == val`. Useful for catching silent state corruption, but only checks liveness at the moment of the check; it won't catch a failure that occurs *during* a subsequent long-running operation.

**Gap to address explicitly: hangs.** A hung Fluent process won't necessarily fail any of the above checks at the moment they're run — it may simply never respond. This needs a timeout/heartbeat mechanism layered on top of (not instead of) the above: if an operation exceeds an expected duration envelope with no liveness signal, treat it as a critical failure and proceed through the same recovery path as an explicit error.

## Failover transparency

Not addressed in the original proposal: when a secondary is promoted to primary (modes 1/3/5), what happens to Python objects the user is already holding — e.g. `solver.settings.some_object` — that are bound to the old primary session? If these break on promotion, the resilience isn't actually transparent to the user's script; it only helps if objects are re-fetched after every command, which defeats much of the ergonomic point.

This likely needs an indirection layer: a stable session handle/proxy that PyFluent objects reference indirectly, so that promotion can be absorbed underneath without invalidating user-held references. Worth scoping as part of this work rather than deferring, since it determines whether "resilience" is genuinely drop-in or requires defensive coding on the user's side.

## Licensing cost

Modes that run a warm secondary session (1, 3, 5) keep a second live Fluent session running for the duration of the primary session, not just during failover. Depending on Fluent's license model, that likely consumes a second license/HPC allocation continuously. This is a real adoption blocker if the feature is positioned as a low-friction, opt-in toggle "any user can configure" — the cost should be quantified and surfaced to users at configuration time (e.g. a warning or an explicit cost estimate when enabling a warm-secondary mode), and single-session modes (2, 4, or mode 5 without a standby) should be the more discoverable default for cost-sensitive users.

## Summary of recommendations

- Treat "mode" as two axes (session redundancy × recovery source), not four flat options.
- Distinguish replayable (settings) vs. non-replayable (solve/mesh) commands explicitly in the design.
- Gate journal/undo-redo-only recovery on single-client sessions; require case+data-based recovery for multi-client sessions.
- Adopt a checkpoint-interval hybrid (mode 5) as the recommended default rather than treating journal-only and case+data-only as the only real choices.
- Layer a timeout/heartbeat mechanism on top of the listed failure-detection signals to catch hangs specifically.
- Scope a session-indirection layer so failover doesn't invalidate user-held object references.
- Quantify and surface the licensing cost of warm-secondary modes at configuration time.
- Explicitly note the undo/redo dependency and treat journal-replay as the interim mechanism until it ships.

No files, tests, or code entry points are named. Start by resolving the proposed resilience axes, recovery fidelity, client-cardinality rules, failure detection, failover object handling, and licensing behavior before locating implementation areas. Done requires an agreed, bounded design and explicit scope rather than the current collection of alternatives.

Written by the indexing model from the issue text.