[Proposal] Support long-running grain calls using a heartbeating mechanism
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
It's common for developers to want long-running grain calls, eg calls which routines run for 10s of seconds or an arbitrarily long time.
Currently, there are two ways to accomplish this, and both have significant detractors:
1. Configure `SiloMessagingOptions.ResponseTimeout` and `ClientMessagingOptions.ResponseTimeout` to some a value large enough to cover the longest call. This has the downside that all calls have the same timeout and therefore they suffer the consequences (eg, deadlocks take longer to unwind, client calls to a failed silo *may* take much longer to fault, leaving the client waiting on a response which will never arrive).
2. Use a polling/callbacks pattern: work is started in the background using one method call and the client either continuously checks for progress using an additional call (maybe a single method is used and invoked repeatedly) or registers a callback. These techniques add complexity to both the grain code and the client code.
### Proposal
We introduce support for arbitrarily long-running calls by leveraging a heart-beating mechanism.
### Mechanism
#6672 added functionality to send updates for long-running requests to callers. The original purpose was to aid in diagnostics for blocked calls (eg, synchronous blocking, slow database connection, request deadlocks). This mechanism can be used to extend the timeout for a request. It could be limited to messages which are allowed to be long-running (i.e., dependent on a policy)
### Policy
We can consider the policy for long-running calls separately. Here are some options, which aren't all mutually exclusive:
1. All calls are always allowed to be long-running
1. A global configuration allows all calls to be long-running
1. Interleaved/reentrant calls are allowed to be long-running
1. Grains can request additional time to process a call programmatically. It would seem that `RequestContext` is the appropriate place to interact with such functionality, but that's TBD
1. Grain interface methods marked with a `[LongRunning]` attribute are allowed to be long-running. In terms of implementation, this could result in an `InvokeMethodOptions` value being set which tells the caller to extend the timeout when a status update is received. This is likely the easiest conservative design
### Considerations
* How should this interact with message expiration? One approach would be to prevent expiry if a message has reached the invocation stage, i.e, do not drop responses for `[LongRunning]` messages which have expired. We could extend that to all messages (not just `[LongRunning]`) with little/no negative effect.
* We do not currently have a **general-purpose cancellation mechanism** for calls. We have support for `GrainCancellationToken`, but that is not ideal: it is not pay-for-what-you use and currently requires scanning all arguments, which therefore boxes every argument in order to perform the type check. It was implemented with what was available at the time, but we have completely rewritten our RPC system since then and could add support for grain cancellation tokens. It seems that this proposal touches on the same area, since it's about timeouts (cancellation, essentially - would it be useful to cancel abandoned calls automatically?). Using `RequestContext` to flow a `CancellationToken` down the call chain seems appropriate. We could also support having a single `CancellationToken` argument in grain interface method signatures. That could then be flowed to the `RequestContext`, but it's obvious that there is room for gotchas, like when both an `CancellationToken` argument and a `RequestContext.Canceled` token is present, but the former isn't flowed by the developer down the chain and the latter is implicitly flowed. We should expand in another issue/proposal.
* How should this work for `[OneWay]` calls? Currently, we do not register `[OneWay]` calls with the local callback collection, so they are entirely untracked by the caller (since timeouts are not possible). If there is a programmatic way to extend call duration, that information would be ignored by the caller. A similar issue exists, for cancellation: if cancellation is lazy (eg, waits until the next heartbeat) then there would be no rendezvous point.
### Related issues
* #4328
Contributor guide
Research direction
Start by reading the status-update mechanism from #6672 and the related proposal in #4328. Then inspect SiloMessagingOptions.ResponseTimeout, ClientMessagingOptions.ResponseTimeout, RequestContext, GrainCancellationToken, and OneWay call behavior. Done requires an agreed policy and implementation covering heartbeats, expiration, cancellation, and long-running call eligibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100