elastic / elastic/apm-agent-php

Make async backend queue size configurable

Open
#1,441 0 comments 1 reaction 0 assignees View on GitHub
agent-php enhancement
Dominant language
PHP
Stars
268
Forks
86
PR merge metrics
No merged PRs in 30d

Description

# Make async backend queue size configurable

## Summary

The PHP agent's async backend communication appears to use a fixed internal queue size with no public configuration option to increase it.

In our CLI workload, events are produced faster than the background sender can flush them to APM Server. Once that queue fills up, later events are rejected locally and never arrive at the server. In agent logs we see:

```text
[ERROR] [Backend-Comm] [backend_comm.cpp:1187] [enqueueEventsToSendToApmServer] Already queued events are above max queue
[ERROR] [Backend-Comm] [backend_comm.cpp:1220] [enqueueEventsToSendToApmServer] Exiting...; resultCode: resultFailure (6)
[ERROR] [Ext-API] [elastic_apm_API.cpp:258] [elasticApmSendToServer] Exiting...; resultCode: resultFailure (6)
```

From reading the source, this seems related to the internal queue size constant `ELASTIC_APM_MAX_QUEUE_SIZE_IN_BYTES` in `backend_comm.cpp`.

## Environment

- Ubuntu on WSL2
- PHP 7.3
- Elastic APM PHP agent 1.15.1
- APM Server 8.12.1
- CLI SAPI
- `async_backend_comm = true`
- `transaction_max_spans = 10000`
- Single long-running CLI process handling about 600 transactions in one run

## Workload

Each loop iteration creates one transaction with custom spans.

The extension also auto-instruments many DB calls for us, so each transaction generates a relatively large number of events.

The configured span limit is `transaction_max_spans = 10000`, but the actual observed span count is much lower than that. In practice, a single transaction is usually around 1000 spans, give or take, and a single CLI process run handles about 600 such transactions.

## Observed behavior

- With `async_backend_comm = true`, only the earlier transactions in the process are delivered reliably.
- After the internal queue fills up, later events are rejected and do not reach APM Server.
- In our tests, only about 40% of transactions arrived when processing a large batch in one process.
- With `async_backend_comm = false`, all transactions are delivered, but total runtime is about 2x slower.

## Expected behavior

There should be a supported way to tune the async send queue/buffer size for workloads that produce bursts of APM events, especially for CLI jobs and other long-running processes.

## Request

Please add a public configuration option for the async backend queue size, for example:

```ini
elastic_apm.max_send_queue_size = 100MB
```

The exact option name and units are up to you. The important part is having a supported way to raise the queue limit when the default is too small for a given workload.

If a configurable byte-based queue is not the preferred design, an equivalent supported setting would still solve the operational problem.

## Why this matters

Right now the only practical workaround we found is:

```ini
elastic_apm.async_backend_comm = false
```

That avoids the queue entirely, but it also makes the CLI job significantly slower and couples job latency to APM Server responsiveness.

Other Elastic APM agents already expose queue/buffer sizing controls. For example, Java has `max_queue_size` and Go has `ELASTIC_APM_API_BUFFER_SIZE`. I could not find an equivalent setting in the PHP agent documentation.

## Additional context

- This seems related to, but not the same as, [#836](https://github.com/elastic/apm-agent-php/issues/836).
- If useful, I can provide a reduced reproduction script and full debug logs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.