googleapis / googleapis/google-cloud-node

pubsub: concurrent ACK/modACK queue additions exceed maxMessages in 6.0.1

Open
#9,303 0 comments 0 reactions 0 assignees View on GitHub
api: pubsub
Dominant language
TypeScript
Stars
3.2k
Forks
712
Avg merge
2d 3h
Merged PRs (30d)
99

Description

### Please make sure you have searched for information in the following guides.

- [x] Search the issues already opened: https://github.com/GoogleCloudPlatform/google-cloud-node/issues
- [x] Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js
- [x] Check our Troubleshooting guide: https://github.com/googleapis/google-cloud-node/blob/main/docs/troubleshooting.md
- [x] Check our FAQ: https://github.com/googleapis/google-cloud-node/blob/main/docs/faq.md
- [x] Check our libraries HOW-TO: https://github.com/googleapis/gax-nodejs/blob/main/client-libraries.md
- [x] Check out our authentication guide: https://github.com/googleapis/google-auth-library-nodejs
- [x] Check out handwritten samples for many of our APIs: https://github.com/GoogleCloudPlatform/nodejs-docs-samples
- [x] Check the API's issue tracker: https://cloud.google.com/support/docs/issue-trackers

### Library Name

@google-cloud/pubsub 6.0.1

### A screenshot that you have tested with "Try this API".

Not applicable: this reproduction isolates a client-side batching problem before requests reach the Pub/Sub API.

It uses the installed SDK’s real subscriber, messages, and acknowledgement queues. Only the Google RPC boundary is replaced with a recorder. No credentials, cloud resources, or API calls are required.

### Link to the code that reproduces this issue. A link to a **public** Github Repository or gist with a minimal reproduction.

https://gist.github.com/vienneraphael/0148221c957167d7adf033c391df2ba4

### A step-by-step description of how to reproduce the issue, based on the linked reproduction.

1. Download reproduce.cjs from the linked gist into an empty directory.
2. Install the unpatched published package:

```sh
npm init -y
npm install --ignore-scripts --no-audit --no-fund @google-cloud/pubsub@6.0.1
```

3. Run:

```sh
node reproduce.cjs
```

4. Observe the JSON output. Sequential additions respect the configured limit, while concurrent additions exceed it for both AckQueue and ModAckQueue.

Environment: Node.js 22.22.2, npm 10.9.7, macOS 26.6.2 (ARM64). We downloaded the published gist into a fresh directory and verified the results with a clean npm installation.

The script deliberately exercises internal queue classes to isolate this behavior. It asserts that every requested ACK ID is recorded exactly once and reports batching-limit violations through respectsLimit: false.

### A clear and concise description of what the bug is, and what you expected to happen.

Concurrent calls to MessageQueue.add() can produce acknowledgement and modify-ack-deadline RPC batches containing more ACK IDs than the configured maxMessages.

**Expected:** Each outgoing RPC contains at most maxMessages ACK IDs, including when additions overlap.

**Actual:**

| Queue | Additions | maxMessages | Messages added | Recorded RPC batch sizes |
|---|---|---:|---:|---|
| AckQueue | Sequential | 1 | 3 | [0, 1, 0, 1, 0, 1] |
| AckQueue | Concurrent | 1 | 3 | [0, 0, 0, 3] |
| AckQueue | Concurrent | 2 | 6 | [1, 1, 1, 3] |
| ModAckQueue | Sequential | 1 | 3 | [1, 1, 1] |
| ModAckQueue | Concurrent | 1 | 3 | [3] |
| ModAckQueue | Concurrent | 2 | 6 | [1, 1, 1, 3] |

All four concurrent cases exceed the configured limit. Both sequential controls pass.

The ACK queue also invokes the recorded RPC with empty batches in some cases; the primary issue here is exceeding the maximum batch count.

### A clear and concise description WHY you expect this behavior, i.e., was it a recent change, there is documentation that points to this behavior, etc. **

[MessageQueue.add()](https://github.com/googleapis/google-cloud-node/blob/pubsub-v6.0.1/handwritten/pubsub/src/message-queues.ts) explicitly checks the configured message-count and byte limits before adding a message, flushing the existing batch when necessary. This indicates the limits are intended to constrain outgoing RPC batches.

The suspected race is between that capacity check and insertion:

1. A call checks the current batch size.
2. It reaches `await this.flush(reason)` before appending its message.
3. Other concurrent calls can reach the same point.
4. The calls resume and append to the shared queue without rechecking its capacity.

This appears to explain why sequential additions respect the limit while concurrent additions do not. In particular, setting `maxMessages: 1` does not reliably produce one ACK ID per RPC.

### Related issue and prior fix

This is related to [googleapis/nodejs-pubsub#2026](https://github.com/googleapis/nodejs-pubsub/issues/2026), addressed by [googleapis/nodejs-pubsub#2027](https://github.com/googleapis/nodejs-pubsub/pull/2027). That issue concerned failed ACK requests being re-queued by `MessageQueue.handleRetry()` without respecting batch limits.

The reproduction here exercises concurrent **initial additions** through `MessageQueue.add()`, with every recorded RPC succeeding and no retries. It still exceeds `maxMessages` on a clean, unpatched 6.0.1 installation. The prior retry-path fix therefore does not cover this reproduction.

Contributor guide

Open the contributing guide

Research direction

Start with handwritten/pubsub/src/message-queues.ts, especially MessageQueue.add(), and run the linked reproduce.cjs script against @google-cloud/pubsub 6.0.1 to observe the concurrent AckQueue and ModAckQueue batches. Compare the concurrent and sequential results, then verify that concurrent additions keep every outgoing RPC at or below maxMessages without losing ACK IDs.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.