Make Hot-Reload Transactional with Reliable Last-Known-Good Rollback

# Design: Transactional Hot-Reload and Reliable Last-Known-Good Rollback

## Status

Proposed design

## How to Read the Source References

Links under **Current code touchpoints** identify existing source files whose current behavior is being described and which will likely need to be refactored during implementation. They are not example implementations of the proposed design.

The proposed classes, interfaces, algorithms, and code fragments in this document describe the new implementation. The linked source files show where the current behavior lives so the future implementation can be grounded in the existing architecture.

---

## Problem Statement

The current hot-reload pipeline treats a configuration as last-known-good as soon as configuration validation succeeds, before all dependent runtime services have successfully refreshed.

The current sequence is approximately:

1. Parse the new configuration.
2. Replace the loader's active `RuntimeConfig`.
3. Validate the new configuration.
4. Save it as `LastValidRuntimeConfig`.
5. Raise ordered hot-reload events.
6. Each dependent service mutates its live state.

### Current code touchpoints

- [FileSystemRuntimeConfigLoader.cs](src/Config/FileSystemRuntimeConfigLoader.cs) loads a new configuration and begins the hot-reload sequence.
- [RuntimeConfigLoader.cs](src/Config/RuntimeConfigLoader.cs) stores active/LKG state and raises the ordered component events.
- [RuntimeConfigProvider.cs](src/Core/Configurations/RuntimeConfigProvider.cs) validates the newly loaded configuration and currently marks it as LKG before component refresh finishes.
- [HotReloadEventHandler.cs](src/Config/HotReloadEventHandler.cs) dispatches the existing string-named refresh events.

This creates two problems:

1. A dependent service can fail after the new configuration has already become the LKG.
2. Services that refreshed before the failure are not restored to their previous state.

`RestoreLkgConfig()` restores only the configuration reference. It does not restore metadata providers, authorization maps, query engines, GraphQL schemas, or other cached state.

---

## Example Failure

Assume configuration A is active and fully functional.

Configuration B:

- passes ordinary runtime configuration validation;
- changes one or more entities;
- causes a later runtime component to fail while rebuilding.

An MCP-specific example would be two entities that normalize to the same tool name, such as `ReadRecords` colliding with the built-in `read_records` tool.

The existing sequence can result in:

- active `RuntimeConfig`: B;
- `LastValidRuntimeConfig`: B;
- query managers: B;
- metadata providers: B;
- authorization state: B;
- MCP registry: A because its refresh failed;
- other components: potentially either A or B.

If the next edit is malformed, restoring the LKG restores B rather than the last completely operational configuration A.

This is a general hot-reload problem, not an MCP-specific problem.

---

## Goals

1. Treat a configuration as active only after every critical runtime component can apply it.
2. Define LKG as the last configuration successfully applied to the entire runtime.
3. Build replacement component state without mutating live state.
4. Publish prepared state only after every participant has prepared successfully.
5. Restore all already-committed components if an unexpected commit failure occurs.
6. Keep the previous runtime completely usable when parsing, validation, or preparation fails.
7. Serialize overlapping file-change notifications.
8. Signal configuration change tokens only after a successful commit.
9. Distinguish critical transactional work from noncritical post-commit notifications.
10. Produce clear structured logs for each reload generation and failure stage.

---

## Non-Goals

The initial implementation does not need to provide:

- dynamic ASP.NET endpoint mapping;
- hot-reload of endpoint paths;
- hot-enabling services that were not registered at startup;
- perfect generation isolation for requests already in flight;
- rollback of external mutations performed outside DAB;
- persistence of LKG state across process restarts.

Settings that require endpoint or middleware reconstruction must be rejected as restart-required changes during hot-reload.

---

## Required Invariants

After a hot-reload attempt completes, exactly one of these must be true:

### Successful reload

- active configuration is the candidate;
- LKG is the candidate;
- every critical component contains candidate state;
- change-token subscribers are notified;
- noncritical post-commit observers are invoked.

### Failed reload

- active configuration remains the previous configuration;
- LKG remains the previous configuration;
- every critical component contains previous state;
- candidate resources are disposed;
- no configuration change token is signaled.

A configuration that merely passes schema validation must not become LKG.

---

## Proposed Architecture

### 1. Separate Candidate Configuration From Active Configuration

Loading a file for hot-reload must produce a candidate without immediately replacing the active configuration.

Introduce an immutable value such as:

```csharp
public sealed record RuntimeConfigCandidate(
long Generation,
RuntimeConfig Config,
string SourcePath,
DateTimeOffset DetectedAt);
```

Add a candidate-loading API to `RuntimeConfigLoader`, for example:

```csharp
public bool TryLoadCandidate(
string path,
[NotNullWhen(true)] out RuntimeConfigCandidate? candidate,
bool isDevelopmentMode,
DeserializationVariableReplacementSettings replacementSettings);
```

Candidate loading may:

- read the file;
- deserialize it;
- replace environment and Key Vault references;
- preserve startup-bound values where required;
- perform parse-level validation.

Candidate loading must not:

- replace the active `RuntimeConfig`;
- modify the LKG;
- signal a change token;
- invoke component refresh events.

Remove the hot-reload dependence on:

- `IsNewConfigDetected`;
- `IsNewConfigValidated`;
- `DoesConfigNeedValidation()`.

These flags exist because the current loader mutates active state before validation. They should no longer be necessary once candidates are isolated.

### 2. Make `RuntimeConfigProvider` the Active State Owner

`RuntimeConfigProvider` is documented as the owner of active runtime configuration, but the mutable state currently lives primarily in `RuntimeConfigLoader`.

Move active-state publication behind `RuntimeConfigProvider`.

Suggested state:

```csharp
private RuntimeConfig _activeConfig;
private RuntimeConfig _lastAppliedConfig;
private long _activeGeneration;
```

Expose internal operations such as:

```csharp
internal void CommitCandidate(RuntimeConfigCandidate candidate);
internal RuntimeConfigSnapshot GetActiveSnapshot();
```

`GetConfig()` and `TryGetConfig()` must return only committed state.

Use atomic reference publication, such as `Volatile.Read()` and `Interlocked.Exchange()`, where appropriate.

Rename `LastValidRuntimeConfig` to `LastAppliedRuntimeConfig` or equivalent. “Valid” is ambiguous because schema-valid does not mean successfully applied.

### 3. Introduce a Hot-Reload Coordinator

Add a singleton service in Core:

```csharp
public sealed class RuntimeConfigHotReloadCoordinator
```

Responsibilities:

1. Serialize reload attempts.
2. Load or receive the latest candidate.
3. Validate hot-reload compatibility.
4. Validate the candidate configuration.
5. Prepare every transactional participant.
6. Commit every prepared change.
7. Publish the candidate through `RuntimeConfigProvider`.
8. Update the LKG.
9. Signal configuration change tokens.
10. Run post-commit observers.
11. Roll back committed participants on unexpected commit failure.
12. Dispose candidate or retired state correctly.

Use a `SemaphoreSlim` or a single-reader `Channel` to prevent concurrent reload transactions.

The file-watcher callback should enqueue a reload request and return quickly. It should not perform database initialization or other long-running work directly on the file-watcher thread.

Multiple notifications for the same file save should be coalesced. If a new notification arrives during a reload, process the latest file contents once the current attempt finishes.

### 4. Add Transactional Participant Contracts

Replace critical string-based hot-reload events with explicit transactional participants.

Suggested contracts:

```csharp
public interface IHotReloadParticipant
{
string Name { get; }

HotReloadStage Stage { get; }

ValueTask PrepareAsync(
HotReloadContext context,
CancellationToken cancellationToken);
}

public interface IPreparedHotReloadChange : IAsyncDisposable
{
void Commit();

void Rollback();
}
```

`PrepareAsync()` may perform:

- validation;
- database metadata discovery;
- construction of replacement dictionaries;
- construction of replacement clients or engines;
- schema generation;
- any operation that can reasonably fail.

`PrepareAsync()` must not mutate live service state.

`Commit()` must:

- perform only in-memory publication;
- preferably be an `Interlocked.Exchange()` or equivalent reference swap;
- perform no file, database, or network I/O;
- be designed not to throw.

`Rollback()` must:

- restore the exact previous state captured before commit;
- perform only in-memory publication;
- be safe when called after a partial commit sequence.

Resource ownership rules:

1. Before commit, the prepared change owns candidate resources.
2. After commit, the live service owns candidate resources.
3. Previous resources must remain available until the complete transaction succeeds.
4. After successful completion, previous resources may be disposed.
5. After failed preparation, candidate resources must be disposed.
6. After rollback, candidate resources must be disposed and previous resources retained.

### 5. Define Explicit Reload Stages

Replace ordering hidden in event invocation order with a typed stage model.

Suggested initial stages:

```csharp
public enum HotReloadStage
{
QueryManagers = 100,
MetadataProviders = 200,
QueryEngines = 300,
MutationEngines = 400,
Documentation = 500,
Authorization = 600,
McpRegistry = 700,
GraphQLSchema = 800,
AuthenticationOptions = 900,
Logging = 1000
}
```

The coordinator prepares participants in stage order.

Rollback occurs in reverse commit order.

Participants in the same stage should run sequentially initially. Parallel preparation can be considered later only when dependencies are explicitly understood.

### 6. Add a Transaction-Local Preparation Context

Some candidate components depend on other candidate components. For example, candidate metadata providers may require candidate query-manager state.

`HotReloadContext` should contain:

```csharp
public sealed class HotReloadContext
{
public RuntimeConfigSnapshot Current { get; }

public RuntimeConfigCandidate Candidate { get; }

public HotReloadPreparedState PreparedState { get; }
}
```

`HotReloadPreparedState` may provide typed transaction-local state:

```csharp
public void Add(T state);
public T GetRequired();
```

This is not application DI. It exists only for one reload transaction and allows later preparation stages to consume state prepared by earlier stages without publishing it globally.

Dependencies between stages must be documented and covered by tests.

---

## Transaction Workflow

### Phase 1: Detect and Load

1. File watcher detects a change.
2. Reload request is queued.
3. Coordinator takes the reload serialization lock.
4. Loader reads the newest file contents.
5. Loader creates a `RuntimeConfigCandidate`.
6. Active configuration remains unchanged.

### Phase 2: Compatibility Validation

Compare the current and candidate configurations.

Reject changes to settings that cannot safely reload, such as:

- endpoint paths;
- middleware shape;
- initially unregistered services;
- host mode;
- other startup-bound settings discovered during implementation.

Return a clear restart-required error rather than accepting a configuration that does not match the running endpoint table.

Create a dedicated component such as:

```csharp
public sealed class HotReloadCompatibilityValidator
```

The validator should report every incompatible property in one result where practical.

### Phase 3: Candidate Validation

Validate the candidate directly.

Add candidate-aware validation APIs so `RuntimeConfigValidator` does not need the candidate to become globally active before validating it.

Separate validation into:

- static configuration validation;
- runtime preparation validation.

Validation that requires constructing metadata providers or contacting a database belongs in participant preparation.

Do not call `SetLkgConfig()` during this phase.

### Phase 4: Prepare

For each participant in stage order:

1. Call `PrepareAsync()`.
2. Record the returned prepared change.
3. Make any required prepared state available to later stages.

If a participant fails:

1. Stop preparation.
2. Dispose all prepared changes in reverse order.
3. Keep active configuration and live services unchanged.
4. Log the participant and exception.
5. Do not signal change tokens.

### Phase 5: Commit

Once every participant has prepared successfully:

1. Enter the short commit section.
2. Call `Commit()` for each prepared change in stage order.
3. Publish the candidate through `RuntimeConfigProvider`.
4. Set the candidate as `LastAppliedRuntimeConfig`.
5. Increment the active generation.
6. Exit the commit section.

No validation, database work, or schema generation should happen during commit.

### Phase 6: Unexpected Commit Failure

Although commits must be designed not to throw, handle unexpected failures defensively.

If commit fails:

1. Do not publish the candidate configuration.
2. Call `Rollback()` on already-committed changes in reverse order.
3. Dispose candidate state.
4. Keep the previous configuration and LKG.
5. Log a critical transaction failure.

If rollback itself fails, the runtime state is indeterminate. In that case:

- report unhealthy status;
- log at critical severity;
- stop the application gracefully rather than continue with unknown mixed state.

### Phase 7: Post-Commit

Only after the transaction succeeds:

1. Signal `RuntimeConfigProvider` change tokens.
2. Refresh `IOptionsMonitor` consumers.
3. Send MCP list-change notifications.
4. Emit informational logs and telemetry.
5. Dispose retired previous-generation resources.

Post-commit observer failure must be logged but must not roll back an already committed runtime. Any observer required for correctness belongs in the transactional participant set instead.

---

## Change-Token Semantics

`RuntimeConfigProvider.GetChangeToken()` currently signals before the ordered component refresh pipeline completes.

Change this behavior so that the token means:

> A complete new runtime generation was successfully committed.

The token must not mean merely:

> A new file was detected or parsed.

This ensures JWT options and other change-token consumers never observe a candidate that later fails to apply.

---

## Components to Convert

### Query Manager Factory

Refactor `QueryManagerFactory` so it can build replacement dictionaries without assigning them to live fields.

Prepared state should include:

- query builders;
- query executors;
- database exception parsers.

Commit should atomically swap one containing state object instead of assigning three fields independently.

#### Current code touchpoint

- [QueryManagerFactory.cs](src/Core/Resolvers/Factories/QueryManagerFactory.cs)

### Metadata Provider Factory

Do not clear the live provider dictionary before replacement providers initialize.

Preparation should:

1. Build a separate provider dictionary.
2. Initialize all providers.
3. Collect initialization errors.
4. Fail without touching live providers if initialization fails.

Commit should swap the complete provider snapshot.

#### Current code touchpoint

- [MetadataProviderFactory.cs](src/Core/Services/MetadataProviders/MetadataProviderFactory.cs)

### Query and Mutation Engine Factories

Build complete replacement factory state during preparation.

Commit each complete state with one reference swap.

#### Current code touchpoints

- [QueryEngineFactory.cs](src/Core/Resolvers/Factories/QueryEngineFactory.cs)
- [MutationEngineFactory.cs](src/Core/Resolvers/Factories/MutationEngineFactory.cs)

### Authorization Resolver

Build the candidate permission map and explicit-role set separately.

Publish both inside one immutable authorization snapshot.

#### Current code touchpoint

- [AuthorizationResolver.cs](src/Core/Authorization/AuthorizationResolver.cs)

### OpenAPI Documentor

Generate candidate documentation without replacing the currently served document.

Publish only during commit.

#### Current code touchpoint

- [OpenApiDocumentor.cs](src/Core/Services/OpenAPI/OpenApiDocumentor.cs)

### GraphQL Schema

Combine eviction, creation, and refresh into one transactional participant.

Preparation should build and validate the replacement schema without evicting the live schema.

Commit should publish the prepared schema or executor.

Do not evict the active schema until the replacement is known to be valid.

#### Current code touchpoints

- [GraphQLSchemaCreator.cs](src/Core/Services/GraphQLSchemaCreator.cs)
- [Startup.cs](src/Service/Startup.cs)

### Logging

Parse and validate candidate log-level configuration during preparation.

Apply it after critical runtime state commits.

Preserve existing production behavior where only supported logging changes are honored.

### Authentication Options

Build and validate candidate JWT/authentication options before commit.

Signal the options change token only after the runtime transaction succeeds.

#### Current code touchpoint

- [JwtBearerOptionsChangeTokenSource.cs](src/Service/JwtBearerOptionsChangeTokenSource.cs)

### MCP Tool Registry

The MCP registry should become a participant once MCP registry hot-reload is implemented.

Preparation should build and validate a complete registry snapshot.

Commit should atomically swap the snapshot.

A failed registry preparation must leave the previous registry active and cause the complete candidate reload to be rejected.

### Other Cached Services

Before implementation, inventory every type that:

- subscribes to `HotReloadEventHandler`;
- subscribes to `RuntimeConfigLoadedHandlers`;
- subscribes to the runtime change token;
- caches values from `RuntimeConfig`;
- is constructed conditionally from startup configuration.

For each type, classify it as:

1. transactional participant;
2. post-commit observer;
3. direct dynamic reader requiring no refresh;
4. startup-bound setting requiring restart;
5. unsupported hot-reload behavior that must be documented.

This inventory is an explicit deliverable of the implementation.

---

## Dependency Injection Registration

A participant must use the same singleton instance as the live service.

Avoid registering separate instances for the service interface and participant interface.

Use a pattern equivalent to:

```csharp
services.AddSingleton();

services.AddSingleton(
provider => provider.GetRequiredService());

services.AddSingleton(
provider => provider.GetRequiredService());
```

Alternatively, register a lightweight participant adapter that targets the existing singleton.

---

## Handling Requests During Commit

Preparation may take time, but it does not affect live state.

The commit section should be very short because it consists only of reference swaps.

Optionally add a `HotReloadCommitBarrier`:

- requests beginning during the commit window wait for commit or rollback to finish;
- requests already in flight may finish using state they already resolved;
- the barrier is released after either successful commit or completed rollback.

Perfect per-request generation isolation is not required for the initial implementation, but no partial state may remain after the transaction finishes.

---

## File-Watcher and Reload Queue Behavior

Implement a hosted reload-processing service or equivalent single-reader queue.

Required behavior:

- file-watcher callbacks enqueue work only;
- duplicate notifications are coalesced;
- only one reload transaction runs at a time;
- if another change arrives while processing, rerun against the newest file;
- shutdown cancels queued work;
- reload exceptions never terminate the file-watcher callback thread;
- every attempt receives a generation or correlation ID.

---

## Late Configuration

The configuration endpoint uses `RuntimeConfigLoadedHandlers`, which has similar partial-application behavior.

The first implementation may focus on file hot-reload, but the long-term design should route late configuration through the same coordinator.

The coordinator must support:

- no active configuration, for first-time late configuration;
- an existing active configuration, for replacement;
- the same prepare/commit guarantees in both cases.

Do not maintain two independent application pipelines indefinitely.

---

## Existing Event System

`HotReloadEventHandler` may remain temporarily for compatibility, but critical state mutation must move to transactional participants.

During migration:

- critical events must not run before commit;
- noncritical events may be invoked as post-commit observers;
- event exceptions must be isolated and logged;
- string event names should eventually be removed for transactional components.

Do not mix old mutating event handlers with new participants for the same component.

---

## Logging and Telemetry

Log one structured record for each stage:

- candidate detected;
- candidate parsed;
- compatibility validation passed or failed;
- configuration validation passed or failed;
- participant preparation started;
- participant preparation completed;
- participant preparation failed;
- transaction commit started;
- transaction committed;
- rollback started;
- rollback completed;
- post-commit observer failed.

Include:

- reload generation;
- configuration path;
- participant name;
- stage;
- elapsed time;
- previous active generation;
- candidate generation;
- failure exception;
- final transaction result.

Expose the latest reload status to health diagnostics where practical.

---

## Failure Semantics

| Failure stage | Active config | LKG | Component state | Change token |
|---|---|---|---|---|
| File read or parse | Previous | Previous | Previous | Not signaled |
| Compatibility validation | Previous | Previous | Previous | Not signaled |
| Config validation | Previous | Previous | Previous | Not signaled |
| Participant preparation | Previous | Previous | Previous | Not signaled |
| Commit with successful rollback | Previous | Previous | Previous | Not signaled |
| Commit with failed rollback | Indeterminate; stop app | Previous | Indeterminate | Not signaled |
| Post-commit observer | Candidate | Candidate | Candidate | Already signaled |

---

## Testing Plan

### Candidate Loading Tests

- Parsing a candidate does not replace active configuration.
- Invalid JSON leaves active configuration and LKG unchanged.
- Environment-variable replacement occurs in the candidate only.
- Production-mode filtering creates an effective candidate without mutating active state.

### Coordinator Tests

- Participants prepare in stage order.
- Participants commit in stage order.
- Preparation failure prevents every commit.
- Prepared changes are disposed in reverse order after failure.
- Commit failure rolls back committed participants in reverse order.
- Active config is published only after all participant commits.
- LKG updates only after complete success.
- Change token fires exactly once after success.
- Change token does not fire after any failure.
- Post-commit failure does not roll back committed state.
- Concurrent reload requests are serialized.
- Duplicate file events are coalesced.

### Participant Tests

For each participant:

- preparation does not mutate live state;
- successful commit publishes all related fields atomically;
- rollback restores the exact prior snapshot;
- failed preparation disposes candidate resources;
- previous resources are retained until transaction completion.

### Integration Tests

1. Start with configuration A.
2. Hot-reload valid configuration B.
3. Verify every runtime component reflects B.
4. Hot-reload malformed configuration C.
5. Verify every runtime component still reflects B.
6. Hot-reload schema-valid configuration D that causes a participant preparation failure.
7. Verify every runtime component and LKG still reflect B.
8. Correct D and verify the next reload succeeds.
9. Trigger rapid consecutive file writes and verify the final file wins.
10. Verify restart-required property changes are rejected without altering active state.

### Fault-Injection Tests

Add test participants that fail during:

- preparation;
- first commit;
- middle commit;
- rollback;
- disposal;
- post-commit notification.

Use these tests to prove coordinator state-machine behavior independently of databases.

---

## Implementation Order

1. Add characterization tests demonstrating the existing LKG/partial-state problem.
2. Add candidate-loading APIs that do not mutate active state.
3. Move active configuration ownership into `RuntimeConfigProvider`.
4. Add `HotReloadCompatibilityValidator`.
5. Add participant and prepared-change contracts.
6. Add the serialized coordinator and reload queue.
7. Convert query-manager state.
8. Convert metadata-provider state.
9. Convert query and mutation engines.
10. Convert authorization state.
11. Convert OpenAPI and GraphQL state.
12. Convert authentication and logging behavior.
13. Move change-token signaling to post-commit.
14. Route file-watcher reloads through the coordinator.
15. Convert or isolate remaining event subscribers.
16. Add MCP registry participation when its hot-reload implementation is available.
17. Route late configuration through the same coordinator.
18. Remove obsolete validation flags and mutating hot-reload event paths.
19. Update the hot-reload design documentation.

Do not switch production hot-reload orchestration to the coordinator until every currently critical event subscriber has either become a participant or been explicitly classified as post-commit/restart-bound.

---

## Acceptance Criteria

The work is complete when:

1. A candidate cannot become active or LKG before all critical participants prepare successfully.
2. A participant preparation failure leaves the complete previous runtime operational.
3. An unexpected commit failure restores every already-committed participant.
4. `RestoreLkgConfig()` is no longer the sole rollback mechanism.
5. The LKG always represents the last fully applied runtime generation.
6. Critical components no longer clear or mutate live state during preparation.
7. Change tokens are emitted only after successful transaction commit.
8. Unsupported startup-bound changes are rejected with a restart-required diagnostic.
9. Overlapping file notifications cannot run overlapping transactions.
10. Integration and fault-injection tests cover parse, validation, preparation, commit, rollback, and post-commit failures.

---

## Key Design Principle

Hot-reload must be treated as a state transition:

`Current Generation -> Prepared Candidate -> Committed Generation`

It must not be treated as a sequence of independent mutation callbacks.

Validation determines whether a candidate is structurally acceptable.

Preparation determines whether the complete runtime can support it.

Commit makes it active.

Only a successfully committed generation is last-known-good.