GoogleCloudPlatform / GoogleCloudPlatform/spring-cloud-gcp

Spring Boot Cloud Run - issues with Pub/Sub subscription after deployment

Open
#4,237 1 comment 1 reaction 0 assignees View on GitHub
priority: p3 type: bug
Dominant language
Java
Stars
551
Forks
349
Avg merge
1d 13h
Merged PRs (30d)
14

Description

Hi,

I am having issues with message acknowledgement when deploying a new revision of a Spring Boot application in Cloud Run. The issue occurs when messages are being pulled (refer to image "Creating subscriptions") after deployment of a new instance.

Following are some images displaying the logs from the application of the following case:

1. Publishing a message that will intentionally fail and triggering retries.
2. Deploying a new revision that stops the message from failing
3. New instance started and message getting acknowledged

The issue occurs in step 3 with the error messages `failed to send operations` and `Permanent error invalid ack id message, will not resend`. These error messages started after enabling `exactlyOnceDelivery`.

I enabled this setting to avoid handling and acknowledging the same message twice - this issue seems to be somewhat resolved as it does not happen every time in this case. However, it has been possible for me to replicate after enabling this setting (See last image of logs).

I want to make sure that I do not handle and acknowledge the same message more than once and also fix these errors occurring related to acknowledging.

Logs before deployment with failed message retrying as expected:

Image

Logs after deployment:

Image

Image

Issues with duplicated message pull around deployment of new instances. Both for retrying failed messages and newly published message only when retried/published short after deployment:

Image

Context:
Spring Boot application with Google Cloud Run (min 1 instance / max 5 instances).

Dependencies:
```

com.google.cloud
spring-cloud-gcp-dependencies
6.2.3
pom
import

```

```

com.google.cloud
spring-cloud-gcp-starter-pubsub

```

```

org.springframework.boot
spring-boot-starter-web

```

application.properties:
```
freddy.pubsub.enabled=true
spring.cloud.gcp.pubsub.publisher.enable-message-ordering=true
spring.cloud.gcp.pubsub.publisher.endpoint=europe-west1-pubsub.googleapis.com:443
spring.cloud.gcp.pubsub.subscriber.min-duration-per-ack-extension=300
```

Creating subscriptions:

*Note: This is executed via an endpoint that is called on-demand.

Image

Pulling Pub/Sub messages:
*Note: This is executed once Spring Boot context event `ApplicationReadyEvent` is published.

Image

Topic in Google Cloud:

Image

Subscription in Google Cloud:

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.