googleapis / googleapis/google-cloud-php
Core/Batch: hardcoded 8192-byte msg_receive() buffer causes a CPU-spin poison message when kernel.msgmax is raised above the default
- Dominant language
- PHP
- Stars
- 1.2k
- Forks
- 463
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 145
Description
#### Environment details
- OS: Linux (Debian/Ubuntu), any distro where `kernel.msgmax` is tunable via sysctl
- PHP version: 8.x (not version-specific — this is a kernel/IPC-level issue)
- Package name and version: `google/cloud-core` v1.72.3 (`Core/src/Batch`), used indirectly via `google/cloud-logging`'s `LoggingClient::psrBatchLogger()` / `BatchDaemon`
#### Summary
`Core\Batch\BatchJob::run()` reads from the SysV message queue with a **hardcoded** 8192-byte buffer and without `MSG_NOERROR`:
https://github.com/googleapis/google-cloud-php/blob/720551ece54b2da33f543d716af3262a6cbd97d8/Core/src/Batch/BatchJob.php#L117-L126
```php
if (msg_receive(
$q,
0,
$type,
8192,
$message,
true,
0, // blocking mode
$errorcode
)) {
```
Meanwhile `SysvProcessor::submit()` never checks the message size before calling `msg_send()` — it relies entirely on the *kernel* to reject oversized messages and fall back to the temp-file path:
https://github.com/googleapis/google-cloud-php/blob/720551ece54b2da33f543d716af3262a6cbd97d8/Core/src/Batch/SysvProcessor.php#L47-L81
This makes the library's correctness implicitly dependent on `kernel.msgmax` being ≤ 8192 (the Linux default), a constraint that is undocumented and unenforced anywhere in the code.
#### Steps to reproduce
1. Raise `kernel.msgmax` above 8192 on the host, e.g. `sysctl -w kernel.msgmax=10240` (a plausible, unrelated tuning change — e.g. to accommodate other software's IPC needs on the same box).
2. Run the batch daemon (`vendor/bin/google-cloud-batch daemon`) with a job using `SysvProcessor` (e.g. via `LoggingClient::psrBatchLogger()`).
3. Submit an item whose serialized size is between 8193 bytes and the new `msgmax` (e.g. a moderately large log entry/context payload). `msg_send()` now succeeds as a direct message, where before the tuning change it would have failed and gone through the temp-file fallback.
4. When the worker's `msg_receive()` reaches that message, it fails with `E2BIG` (message larger than the requested 8192-byte buffer) and, critically, **the message is not removed from the queue** (that's the POSIX/Linux semantics of `msgrcv` without `MSG_NOERROR`).
5. `BatchJob::run()`'s `while (true)` loop immediately retries the same blocking `msg_receive()` call, which fails again with `E2BIG` instantly (the syscall fails validation before it would need to block) → tight loop, ~100% CPU in the worker process.
6. The oversized message is now a permanent "poison pill": nothing behind it in the queue can be consumed (FIFO head-of-line blocking), so the queue fills up until producers start hitting `QueueOverflowException`.
#### Impact
- Silent, sustained CPU exhaustion in a background worker process, with **no error surfaced anywhere in the calling application** — from the app's point of view `msg_send()` succeeded, so logging/batch calls appear to work fine. The only visible symptom is anomalous CPU usage on the host, which is hard to correlate back to this library.
- Any operator who raises `kernel.msgmax` above the Linux default of 8192 — for any reason, not necessarily related to this library — can silently trigger this the next time a slightly-larger-than-usual item is submitted. Since `kernel.msgmax` is a machine-wide setting, this can be caused by completely unrelated software sharing the host.
- We hit this in production: a `StackdriverLogger`-driven `psrBatchLogger()` call encoded to somewhere between 8KB and 10KB (an `httpRequest` context payload) became the poison message as soon as we raised `kernel.msgmax` to 10240 for unrelated reasons. `ipcrm`-ing the affected queues was the only way to recover, and the daemon needed a restart since the running workers held stale queue handles.
#### Suggested fixes
Any of these would resolve it; happy to hear which direction you'd prefer:
1. Pass `MSG_NOERROR` to `msg_receive()` so an oversized message gets truncated and dequeued instead of stalling the loop — ideally combined with logging a warning so the drop is at least visible.
2. Don't hardcode `8192`; read the buffer size from `msg_stat_queue()` (`msg_qbytes`), or make it configurable similar to the existing `GOOGLE_CLOUD_SYSV_ID` env var, so `msg_receive()`'s buffer always matches the queue's actual configured capacity instead of assuming the kernel default.
3. At minimum, treat a `false` return from `msg_receive()` with `$errorcode === MSG_E2BIG` as fatal for that specific message — actively drain-and-discard it (e.g. retry with `MSG_NOERROR`) rather than retrying the identical call forever.
4. Document the `kernel.msgmax` ≤ 8192 requirement explicitly (e.g. in `Core/src/Batch`'s docblocks and any Batch daemon setup guide), and/or have `BatchDaemon`/`SysvConfigStorage` validate it at startup and fail fast with a clear message instead of degrading into an unexplained CPU spin under production load.
Related but distinct from #1336 / PR #3176, which addressed the *producer* side stalling on `submit()` when the queue is full. This report is about the *consumer* side (`msg_receive()`) silently spinning on an oversized message — a different failure mode with a different fix.
Contributor guide
Assessment
This issue has not been assessed yet.