Pub/Sub c++ client: How to prevent/reduce ack failures on at least once delivery subscription

Client version: v2.39.0
MaxDeadline = seconds(0) (default value)
MaxDeadlineExtension = seconds(180)
MinDeadlineExtension = seconds(60)
MaxOutstandingMessages = 250
MaxConcurrency = 1

Time to time acks are failing. There is no retry for `at least once` delivery. How to reduce/prevent these failures? It happens 10-20 times a day.

Some of the logged errors [file_name] - [message]

> [NTPubSubLogger] /google-cloud-cpp/google/cloud/pubsub/internal/ack_handler_wrapper.cc:29 - error while trying to ack(), status=UNAVAILABLE: recvmsg:Connection reset by peer

> [NTPubSubLogger] /google-cloud-cpp/google/cloud/pubsub/internal/ack_handler_wrapper.cc:29 - error while trying to ack(), status=UNAVAILABLE: 502:Bad Gateway

It sounds like network issue but I doubt that. It happens every day 10-20 times. I can reproduce it with a 10 minutes of load tests by consuming/processing 300 messages per second.

Messages are ordered by account id in the system. When this happens the messages are stuck on pub/sub for 10-20 minutes because of unacked message. initial deadline=60 seconds, max deadline = 180 seconds. I observed the cpp client logs in debug mode. There is no ModifyAckDeadlineRequest is being made but still messages redelivered between 10-20 minutes later, in fact they should arrive in a few minutes. (That's another issue and maybe not related to the client.)

まず google/cloud/pubsub/internal/ack_handler_wrapper.cc:29 とレポートに記載されたクライアント設定を調べ、次に説明されている負荷テストで失敗を再現します。UNAVAILABLE エラーと再配信に関連する acknowledgement および deadline-extension の動作を追跡します。ack の失敗と再配信の遅延の原因が特定され、それらを軽減または防止する検証済みの方法が得られれば完了です。

索引モデルが issue の本文から書いたものです。