awslabs / awslabs/amazon-sqs-java-messaging-lib

When retrying error on Queue Throttled, client must use a new ReceiveRequestAttemptId.

Open
#213 0 comments 5 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
183
Forks
150
PR merge metrics
No merged PRs in 30d

Description

### Library version
2.0.3

### SQS configuration
fifo_queue = true
content_based_deduplication = false
visibility_timeout_seconds = 3600
receive_wait_time_seconds = 20
fifo_throughput_limit = perMessageGroupId
deduplication_scope = messageGroup
enable_dlq = true
dlq_max_receive_count = 3
dlq_visibility_timeout_seconds = 30
dlq_message_retention_seconds = 1209600

### Reproduction steps
This error occurs when the SQS is on load.
Had multiple interactions with the AWS support engineers. Here are the details provide on the insight based the SQS server logs.
Queue Throttled” responses while receiving messages, and on the follow-up retry, the response becomes “ReceiveRequestAttemptId is invalid”
Client to retry a throttled request, should use a different ReceiveRequestAttemptId for receiving further messages.
There are no network issue nor it is a response with generic error. Instead, the request responded with Queue Throttled and therefore the ReceiveMessage API call cannot be retried with the same ReceiveRequestAttemptId.

Reason for throttling -> SQS service team found out that SQS FIFO queue gets scaled down for the reason that no send message action was performed for the last few minutes. And then, the request arrived to SQS service side and gets throttled.

When retrying error on Queue Throttled, client must use a new ReceiveRequestAttemptId.

### Stack Trace
**Stack trace ERROR ->**

`software.amazon.awssdk.services.sqs.model.SqsException: Value 85d410a1-f61d-4aeb-a92f-a5bfa7e3e523 for parameter ReceiveRequestAttemptId is invalid. Reason: The receiveRequestAttemptId is not valid anymore since the receive call is not retryable. (Service: Sqs, Status Code: 400, Request ID: b254394b-cf1a-52d9-90ac-675c71ae8949)
at s.a.a.c.i.h.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
at s.a.a.c.i.h.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:40)
at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:30)
at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Queue Throttled (Service: Sqs, Status Code: 403, Request ID: 48b7f4e2-911b-5d1e-8d21-aaf52fa843d0)
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Queue Throttled (Service: Sqs, Status Code: 403, Request ID: 0b15c096-5d5f-50e7-a454-9e81e3ec23f7)`

**Followed by WARN, stack trace ->**

`software.amazon.awssdk.services.sqs.model.SqsException: Value 85d410a1-f61d-4aeb-a92f-a5bfa7e3e523 for parameter ReceiveRequestAttemptId is invalid. Reason: The receiveRequestAttemptId is not valid anymore since the receive call is not retryable. (Service: Sqs, Status Code: 400, Request ID: b254394b-cf1a-52d9-90ac-675c71ae8949)
at s.a.a.c.i.h.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
at s.a.a.c.i.h.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:40)
at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:30)
at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Queue Throttled (Service: Sqs, Status Code: 403, Request ID: 48b7f4e2-911b-5d1e-8d21-aaf52fa843d0)
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Queue Throttled (Service: Sqs, Status Code: 403, Request ID: 0b15c096-5d5f-50e7-a454-9e81e3ec23f7)`

Related error is mentioned here as well - https://github.com/aws/aws-sdk-java/issues/2613#issuecomment-882765440

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the retry path represented by RequestPipelineBuilder, TimeoutExceptionHandlingStage, and CombinedResponseHandler, then compare the behavior with the related AWS SDK issue linked in the report. Reproduce the Queue Throttled sequence if possible and verify that a follow-up ReceiveMessage attempt does not reuse the invalid ReceiveRequestAttemptId.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.