Server returns status messages over metadata limit leading to connection drops and cryptic errors

Expected Behavior

If a worker reports an unexpected sequence of commands, a worker should get a gRPC error like this:

io.grpc.StatusRuntimeException: INVALID_ARGUMENT: invalid command sequence: [], command CompleteWorkflowExecution must be the last command.

irrespectively of the size of the number of commands that were sent.

The server should return an error message that is aligned with the announced receiver limit or at least under the default gRPC-over-HTTP/2 header limit.

Actual Behavior

Reproduction:
https://github.com/Spikhalskiy/java-sdk/commit/74e6e3f2af363afd061cd9ec9f2c0d9b31ea5b21#diff-e566b269374183dd0f8a9068200b3c417dff37dce93f64021fe8cd40e71dee21R80

This reproduction generates a large (but manageable and under all the gRPC limits) and incorrect sequence of commands on the workflow task completion
[CompleteWorkflowExecution, RecordMarker <xManyTimes>]
The worker receives RST_STREAM frame instead of the normal server response and gets a connection closed.
io.grpc.StatusRuntimeException: INTERNAL: RST_STREAM closed stream. HTTP/2 error code: INTERNAL_ERROR.
Another manifestation of this problem on smaller sizes is
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$HeaderListSizeException: Header size exceeded max allowed size (10240)
This leads to a cryptic log and there is no way for an application developer to understand that it's an incorrect sequence of commands causing it.

Workaround

Client-side can temporary set a large limit on incoming headers (maxInboundMetadataSize) like it's shown here:
https://github.com/Spikhalskiy/java-sdk/commit/74e6e3f2af363afd061cd9ec9f2c0d9b31ea5b21#diff-dbd68ad479d675fe5c67f7fe0141d1078d9009fb27f6cd52e5ed6c42bf10d46aR179

Root cause analysis

The server uses a raw sequence of commands as a part of the error message. The server doesn't respect either the SDK settings of maxInboundMetadataSize (that should be announced in the SETTINGS_MAX_HEADER_LIST_SIZE field of SETTINGS HTTP/2 frame) or a standard HTTP/2 limit for headers of 8192 bytes.

Proposed solution

The Server should preprocess all the variable-sized portions of error messages to make sure that it fits into

• a header limit announced by the counterpart (better)
• some reasonable default max length that is lower than the default gRPC metadata limit of 8Kb (acceptable)

It's better to receive an error message without specific commands or with a trimmed list than to get a broken connection and no useful error message at all

Related HTTP/2 specs

6.5.2. Defined SETTINGS Parameters
SETTINGS_MAX_HEADER_LIST_SIZE (0x6): This advisory setting informs a
peer of the maximum size of header list that the sender is
prepared to accept, in octets. The value is based on the
uncompressed size of header fields, including the length of the
name and value in octets plus an overhead of 32 octets for each
header field.

  For any given request, a lower limit than what is advertised MAY
  be enforced.  The initial value of this setting is unlimited.

10.5.1. Limits on Header Block Size

A large header block (Section 4.3) can cause an implementation to
commit a large amount of state. Header fields that are critical for
routing can appear toward the end of a header block, which prevents
streaming of header fields to their ultimate destination. This
ordering and other reasons, such as ensuring cache correctness, mean
that an endpoint might need to buffer the entire header block. Since
there is no hard limit to the size of a header block, some endpoints
could be forced to commit a large amount of available memory for
header fields.

An endpoint can use the SETTINGS_MAX_HEADER_LIST_SIZE to advise peers
of limits that might apply on the size of header blocks. This
setting is only advisory, so endpoints MAY choose to send header
blocks that exceed this limit and risk having the request or response
being treated as malformed. This setting is specific to a
connection, so any request or response could encounter a hop with a
lower, unknown limit. An intermediary can attempt to avoid this
problem by passing on values presented by different peers, but they
are not obligated to do so.

A server that receives a larger header block than it is willing to
handle can send an HTTP 431 (Request Header Fields Too Large) status
code [RFC6585]. A client can discard responses that it cannot
process. The header block MUST be processed to ensure a consistent
connection state, unless the connection is closed.

Start by tracing the server path that validates workflow task command sequences and constructs the error containing the raw command list. Use the linked Java SDK reproduction to exercise an oversized invalid sequence and verify that the response remains an actionable gRPC error rather than RST_STREAM or a header-size failure. Done means variable-sized error content is bounded below the applicable metadata limit.