awslabs / awslabs/amazon-sqs-java-temporary-queues-client

Runaway CPU consumption and queue activity

Open
#29 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
99
Forks
27
PR merge metrics
No merged PRs in 30d

Description

Hi. In testing, I keep running into the situation where `AmazonSQSRequester` creates hundreds of temporary queues, with hundreds of messages in each, and ends up in a state it seemingly can't recover from.

It's hard to figure out what's going on without getting way too well acquainted with the source code, but I've tried to produce a simple test case which can reliably reproduce the problem (or **some** problem anyway - hopefully not a different one):

```
@Test
public void test() throws InterruptedException {
AmazonSQSRequesterClientBuilder
.standard()
.withAmazonSQS(AmazonSQSClientBuilder.defaultClient())
.withInternalQueuePrefix("deleteme")
.build();

while (true) Thread.sleep(500);
}
```

In this example, the queue `deleteme` does not yet exist at test start time. If I leave this test running for a while, after a few minutes, it starts spinning away, sucking significant CPU resources, and generating a huge volume of traffic to SQS (and messages sent to SQS) - here's a summarised extract of API activity over a three second period:
```
16:09:45.091 [SQSMessageConsumer-Thread-1] "Action=SendMessage&Version=2012
16:09:45.306 [SQSMessageConsumer-Thread-1] "Action=ListQueueTags&Version=20
16:09:45.518 [SQSMessageConsumer-Thread-1] "Action=SendMessage&Version=2012
16:09:45.558 [AmazonSQSIdleQueueDeletingClient-Thread-2] "Action=TagQueue&V
16:09:45.730 [SQSMessageConsumer-Thread-1] "Action=TagQueue&Version=2012-11
16:09:45.949 [SQSMessageConsumer-Thread-1] "Action=DeleteMessage&Version=20
16:09:46.155 [SQSMessageConsumer-Thread-1] "Action=ReceiveMessage&Version=2
16:09:46.393 [SQSMessageConsumer-Thread-1] "Action=ListQueueTags&Version=20
16:09:46.607 [SQSMessageConsumer-Thread-1] "Action=ListQueueTags&Version=20
16:09:46.818 [SQSMessageConsumer-Thread-1] "Action=SendMessage&Version=2012
16:09:47.026 [SQSMessageConsumer-Thread-1] "Action=DeleteMessage&Version=20
16:09:47.056 [SQSMessageConsumer-Thread-3] "Action=DeleteMessage&Version=20
16:09:47.234 [SQSMessageConsumer-Thread-1] "Action=ReceiveMessage&Version=2
16:09:47.263 [SQSMessageConsumer-Thread-3] "Action=ReceiveMessage&Version=2
16:09:47.679 [SQSMessageConsumer-Thread-1] "Action=ListQueueTags&Version=20
16:09:47.680 [ForkJoinPool.commonPool-worker-1] "Action=ListQueueTags&Versi
16:09:47.890 [SQSMessageConsumer-Thread-1] "Action=ListQueueTags&Version=20
16:09:47.900 [ForkJoinPool.commonPool-worker-1] "Action=ListQueueTags&Versi
```

The start of the runaway process seems to correspond with the first occurrence of `AmazonSQSIdleQueueDeletingClient - Checking all queues begining with prefix` appearing in the logs. (I guess the random `initialDelay` in `IdleQueueSweeper` causes the delay in the first occurrence of the problem.)

I wonder if #28 is not related (is not the cause)?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.