dotnet / dotnet/orleans

Track gateway-forwarded client requests by destination silo

Open
#10,165 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
13h 56m
Merged PRs (30d)
351

Description

When a client sends a request through a gateway, the gateway forwards the request into the silo runtime and the client waits for a response or rejection. If the destination silo becomes unavailable while that gateway-forwarded request is in flight, the client can currently wait until its normal response timeout instead of being completed promptly with `SiloUnavailableException`.

A recent flaky placement test exposed this: the test stopped the silo hosting a `PreferLocalPlacement` activation, then immediately reused the same client grain reference. The test fix waits for liveness stabilization before making the next request, but that only stabilizes the test. It does not mitigate the runtime gap.

Investigation:

- `SiloConnectionMaintainer.SiloStatusChangeNotification` calls `IRuntimeClient.BreakOutstandingMessagesToSilo(updatedSilo)` when a remote silo is declared dead.
- `InsideRuntimeClient.BreakOutstandingMessagesToSilo` walks the silo runtime callback table and completes callbacks whose `CallbackData.Message.TargetSilo` matches the dead silo.
- Calls originated by grains or the hosted silo client are registered in `InsideRuntimeClient.SendRequest`, so those callbacks can be broken promptly.
- Requests originated by external clients are registered in `OutsideRuntimeClient` on the client process. The gateway receives them in `GatewayInboundConnection.OnReceivedMessage`, clears/rewrites routing fields as needed, and forwards them via `MessageCenter.RerouteMessage` / `AddressAndSendMessage`.
- The gateway's per-client state is currently `Gateway.ClientState` (this appears to be the current equivalent of the proposed ClientData-level tracking). It only tracks the connected client socket and queued messages waiting to be sent back to the client. It does not track gateway-forwarded requests after they have been sent to a target silo.
- Because the gateway does not record which destination silo each client request was sent to, a dead-silo notification cannot identify affected client requests and synthesize/send a rejection response back to the client.

Suggested mitigation:

Track gateway-forwarded client requests in the gateway's per-client state (`Gateway.ClientState`, or an extracted/renamed ClientData type if that is preferred). For each non-one-way client request forwarded through the gateway, record enough information to synthesize a response if the selected target silo becomes unavailable:

- client id / `ClientState`
- request `CorrelationId`
- original request `Message` or the minimal fields required by `MessageFactory.CreateRejectionResponse`
- final destination `TargetSilo` selected by placement/addressing

When the gateway's silo observes `SiloStatus.Dead` for a target silo, it should find all in-flight client requests sent to that silo, remove them from tracking, and send a transient rejection (backed by `SiloUnavailableException`) to the client through the existing `ClientState` response path. Normal responses/rejections should also remove the request from tracking to avoid leaks. Entries should be bounded by response timeout/TTL so abandoned requests are cleaned up.

Implementation considerations:

- The final target silo is not known at the moment `GatewayInboundConnection.OnReceivedMessage` receives a client message if placement/directory lookup is needed. Tracking probably needs to hook after `PlacementService.AddressMessage` has set `message.TargetSilo`, or otherwise be integrated into the gateway forwarding path around `MessageCenter.AddressAndSendMessage`.
- Avoid tracking one-way messages unless a cache invalidation response is required.
- Preserve existing response routing semantics, including `ClientsReplyRoutingCache` for client-addressable-object replies.
- The tracking owner should be gateway/per-client state rather than `InsideRuntimeClient`, since these callbacks belong to the external client and are only proxied by the gateway.

Acceptance criteria:

1. A client request forwarded through a gateway to a silo that is later declared dead completes promptly with `SiloUnavailableException` or an equivalent transient rejection, rather than waiting for the full response timeout.
2. Normal responses and rejections clear the in-flight tracking entry.
3. Tracking is cleaned up on client disconnect/drop and on timeout/TTL expiration.
4. A regression test covers a gateway-forwarded request whose final target silo dies before a response is returned.

Contributor guide

Open the contributing guide

Research direction

Start by reading GatewayInboundConnection, Gateway.ClientState, MessageCenter.RerouteMessage/AddressAndSendMessage, and the SiloConnectionMaintainer notification path, then inspect the existing flaky placement test. Trace how final TargetSilo and client responses are handled. Done means dead-silo requests receive prompt transient rejection, normal responses clean up tracking, disconnects and expiry clean up abandoned entries, and a regression test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.