Update API: support not writing history when calling external failure

Is your feature request related to a problem? Please describe.

Using “short update“ is a very important pattern of Update API feature. It acts like an RPC. Usually, the update handler may call an external system to perform an action, if successful, then mutate the workflow states. If calling external fails, then no mutation on the workflow.

Note that this “short update“ may be called frequently with back off retry on failures. In reality, the external system could be unreliable so the failure is out of control.

As the way it works today, Temporal will write the update failure event into history, even this event doesn’t have any value for this use case. Eventually, this failure event will blow up the history very quickly. Hence this will become a scalability issue to use with Temporal.

Describe the solution you'd like
Based on discussion with @mfateev , it was designed this way because throwing away the activity results means the cache would become invalid hence the workflow has to be replayed to rebuild the cache. This will put significant load on the service as every such rejection would require kicking the workflow out of cache and replaying it from the beginning.

An better idea to implement this is to only allow certain way to calling external system -- for example, using local activity only. In GoSDK, the update API will just change the UpdateHandlerOptions to become like this:

UpdateHandlerOptions struct {
    // Validator is an optional (i.e. can be left nil) func with exactly the
    // same type signature as the required update handler func but returning
    // only a single value of type error.  ....
    Validator interface{}

    // PreUpdateLocalActivity is an optional local activity to be execute after the Validator, 
    // If it succeeds, then invoke PreUpdateLocalActivityValidator if exists, or invoke the update handler
    // if it fails then return the error to Update API caller
    // Note that when this is set, the first parameter of update handler must be accepting the result of the local 
    // activity, and then the other parameters from caller
    PreUpdateLocalActivity interface{}

    // PreUpdateLocalActivityValidator is an optional func like Validator to validate the results of local activity
    // same requirement as Validator. 
    // If succeeds then invoke update handler
    // If fails then return error to Update API caller
    PreUpdateLocalActivityValidator interface{}
    
    // PreUpdateLocalActivityOptions is the options for PreUpdateLocalActivity
    PreUpdateLocalActivityOptions LocalActivityOptions
}

This feature will only need to implement in the SDK. And since we only update the UpdateHandlerOptions, it's backward compatible. Maxim has confirmed that this should work, see the thread in the context.

Essentially, Temporal doesn't assign any activityId(which means the state machine needs to change) or history eventId when invoking the local activity. So if it fails, nothing will be changed to the workflow states(cache). If it succeeds, the pass the results to the real update handler.

Describe alternatives you've considered
Another way to workaround is to ask user to call the external system, if succeeds then passing the results when calling Update API. However, user will have to handle a lot more edge cases, especially racing conditions (as when calling external APIs, workflow is also running, and the workflow states could be changed).

Additional context
https://temporalio.slack.com/archives/CTDTU3J4T/p1681187414101309

Start by locating the Go SDK's UpdateHandlerOptions and the update execution path. Review the linked Slack discussion and existing update and local-activity behavior before defining the API shape. Done means a backward-compatible SDK implementation that supports the proposed pre-update local activity flow without writing failure history.