Node health cannot distinguish Alternator coordinator failures reported as InternalServerError

Description

The client currently treats every received HTTP response as TRAFFIC_SUCCESS. This prevents transport failures from being confused with authentication or request errors, but it also means persistent server-side failures do not affect node health.

Classifying every 5xx response as TRAFFIC_FAILURE is unsafe because Alternator reports several different coordinator failures with the same __type:

{
  "__type": "com.amazonaws.dynamodb.v20120810#InternalServerError",
  "message": "Internal server error: ..."
}

Coordinator timeouts, read/write failures, and unexpected internal errors are therefore indistinguishable without parsing the human-readable message.

The relevant Alternator conversion maps coordinator errors other than rate limiting and overload to api_error::internal(...), which is serialized with __type ending in InternalServerError:

• coordinator error conversion
• error type construction
• response serialization

Impact

• A node persistently returning InternalServerError remains active and can reset previous health-failure progress.
• Treating every InternalServerError as a node failure could mark healthy nodes down during cluster-wide overload.
• Removing nodes during overload reduces available capacity and can cause cascading timeouts.
• Retry and health logic cannot determine whether the failure is node-specific, cluster-wide, or non-retryable.
• AWS SDK error handling cannot recover the original Alternator failure category because every response has the same __type.

Required direction

Alternator should expose stable, distinct __type values, or another machine-readable field, for coordinator timeout, read/write failure, overload, and unexpected internal failure.

Until such a signal is available, this client must not classify every InternalServerError as an unconditional node-health failure. A possible fallback is to penalize a node only when the same logical request subsequently succeeds through another coordinator; otherwise the response should remain health-neutral.

Start by reading the linked Alternator conversion and serialization points in alternator/executor_util.cc, alternator/error.hh, and alternator/server.cc, then locate the Java client’s InternalServerError handling and node-health classification. Compare the available response signals with the required coordinator categories and conservative fallback; done means node health no longer treats every InternalServerError as an unconditional failure.