timeouts and return codes from blocking invocations.
- Dominant language
- Scala
- Stars
- 6.8k
- Forks
- 1.2k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 2
Description
Here's a proposal from @rabbah and myself for a change in how we deal with timeouts in blocking invocations.
### Motivation and principles
1) to be consistent with the mantra of "nothing unbounded", we don't want to hold connections for blocking calls open for long periods of time.
2) blocking calls should be used for short-running, latency-sensitive operations, in general. We don't feel the need to support a blocking call for an action that takes 5 minutes to run. We're fine saying "if your action takes 5 minutes, you must use a non-blocking call".
3) We interpret the HTML error code conventions as follows:
- 200 series -- everything is OK
- 400 series -- client or user error
- 500 series -- server error
### Proposed API change
1) We change the definition of a blocking call to include the following constraint: when making a blocking call, your action will only be allowed to run for N seconds (say N = 30). Suppose your action is configured with the default running timeout of 300 seconds. When making a non-blocking call, your action may run for up to 300 seconds. But (CHANGE here!) when making a blocking call, your action will be timed out after min(300,N) (that is, 30) seconds.
2) When making a blocking call, if the system cannot start running your action within S seconds, (say S=20), then we will return a 503 Service Unavailable, meaning the system is overloaded and cannot service your blocking invocation immediately. However, the result will include an activation id, and your action will still run eventually.
3) We will change the error code for when a user actions runs too long and is timed out while running to be '400 Bad Request'.
So to summarize the new error codes for blocking calls:
200 Success
400 Action started running but was killed because it ran too long
503 System could not start running the action within S seconds of the request.
The 200 response will include the result of the activation. the 400 and 503 responses will describe what happened and include an activation id.
This allows a simple implementation in the controller -- after dispatching an invocation, it simply waits N + S + epsilon seconds for an active ack result. If no active ack is received, 503.
Contributor guide
Assessment
This issue has not been assessed yet.