dotnet / dotnet/orleans

[Proposal] Support long-running grain calls using a heartbeating mechanism

Open
#7,649 18 comments 23 reactions 0 assignees View on GitHub
area-grains enhancement
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

It's common for developers to want long-running grain calls, eg calls which routines run for 10s of seconds or an arbitrarily long time.

Currently, there are two ways to accomplish this, and both have significant detractors:

1. Configure `SiloMessagingOptions.ResponseTimeout` and `ClientMessagingOptions.ResponseTimeout` to some a value large enough to cover the longest call. This has the downside that all calls have the same timeout and therefore they suffer the consequences (eg, deadlocks take longer to unwind, client calls to a failed silo *may* take much longer to fault, leaving the client waiting on a response which will never arrive).
2. Use a polling/callbacks pattern: work is started in the background using one method call and the client either continuously checks for progress using an additional call (maybe a single method is used and invoked repeatedly) or registers a callback. These techniques add complexity to both the grain code and the client code.

### Proposal

We introduce support for arbitrarily long-running calls by leveraging a heart-beating mechanism.

### Mechanism

#6672 added functionality to send updates for long-running requests to callers. The original purpose was to aid in diagnostics for blocked calls (eg, synchronous blocking, slow database connection, request deadlocks). This mechanism can be used to extend the timeout for a request. It could be limited to messages which are allowed to be long-running (i.e., dependent on a policy)

### Policy

We can consider the policy for long-running calls separately. Here are some options, which aren't all mutually exclusive:

1. All calls are always allowed to be long-running
1. A global configuration allows all calls to be long-running
1. Interleaved/reentrant calls are allowed to be long-running
1. Grains can request additional time to process a call programmatically. It would seem that `RequestContext` is the appropriate place to interact with such functionality, but that's TBD
1. Grain interface methods marked with a `[LongRunning]` attribute are allowed to be long-running. In terms of implementation, this could result in an `InvokeMethodOptions` value being set which tells the caller to extend the timeout when a status update is received. This is likely the easiest conservative design

### Considerations

* How should this interact with message expiration? One approach would be to prevent expiry if a message has reached the invocation stage, i.e, do not drop responses for `[LongRunning]` messages which have expired. We could extend that to all messages (not just `[LongRunning]`) with little/no negative effect.
* We do not currently have a **general-purpose cancellation mechanism** for calls. We have support for `GrainCancellationToken`, but that is not ideal: it is not pay-for-what-you use and currently requires scanning all arguments, which therefore boxes every argument in order to perform the type check. It was implemented with what was available at the time, but we have completely rewritten our RPC system since then and could add support for grain cancellation tokens. It seems that this proposal touches on the same area, since it's about timeouts (cancellation, essentially - would it be useful to cancel abandoned calls automatically?). Using `RequestContext` to flow a `CancellationToken` down the call chain seems appropriate. We could also support having a single `CancellationToken` argument in grain interface method signatures. That could then be flowed to the `RequestContext`, but it's obvious that there is room for gotchas, like when both an `CancellationToken` argument and a `RequestContext.Canceled` token is present, but the former isn't flowed by the developer down the chain and the latter is implicitly flowed. We should expand in another issue/proposal.
* How should this work for `[OneWay]` calls? Currently, we do not register `[OneWay]` calls with the local callback collection, so they are entirely untracked by the caller (since timeouts are not possible). If there is a programmatic way to extend call duration, that information would be ignored by the caller. A similar issue exists, for cancellation: if cancellation is lazy (eg, waits until the next heartbeat) then there would be no rendezvous point.

### Related issues

* #4328

Contributor guide

Open the contributing guide

Research direction

Start by reading the status-update mechanism from #6672 and the related proposal in #4328. Then inspect SiloMessagingOptions.ResponseTimeout, ClientMessagingOptions.ResponseTimeout, RequestContext, GrainCancellationToken, and OneWay call behavior. Done requires an agreed policy and implementation covering heartbeats, expiration, cancellation, and long-running call eligibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.