elastic / elastic/apm-agent-php

Log a warning when async shutdown stops with queued events still pending

Open
#1,445 0 comments 0 reactions 0 assignees View on GitHub
agent-php enhancement
Dominant language
PHP
Stars
268
Forks
86
PR merge metrics
No merged PRs in 30d

Description

# Log a warning when async shutdown stops with queued events still pending

## Summary

When the PHP agent uses async backend communication, shutdown appears to stop draining the send queue once its drain deadline is reached. In that path, queued events may be left unsent without any dedicated warning in the agent log.

That makes missing transactions and spans difficult to diagnose. Users can see data loss in APM, but there is no clear log signal that shutdown ended before the queue was fully flushed.

## Observed behavior

In a CLI batch workload, some transactions were missing from APM even though we did not see queue-overflow errors in the agent log.

While investigating, I noticed that the async backend shutdown logic seems to exit on timeout via `shouldExitBy` without emitting a warning in that branch:

```c
// backgroundBackendCommThreadFunc_shouldBreakLoop in backend_comm.cpp
if ( compareAbsTimeSpecs( &sharedStateSnapshot->shouldExitBy, &now ) < 0 )
{
*shouldBreakLoop = true;
goto success;
}
```

I may be missing surrounding context in the function, but from reading this branch it looks like shutdown can stop without a user-visible indication that queued data was still pending.

## Why this matters

The queue overflow path already logs a clear error when events are rejected because the queue limit is exceeded.

The shutdown-timeout path is a different failure mode. If the queue is still draining when the process exits, users need a comparable signal in logs so they can distinguish:

- queue overflow during runtime
- shutdown ending before async flush completed
- normal successful drain

Without that signal, the only visible symptom may be missing APM data after process exit.

## Request

Please log a warning when async shutdown stops because the drain timeout/deadline is reached and there is still queued data waiting to be sent.

Even a simple warning would help, for example:

```c
ELASTIC_APM_LOG_WARNING(
"Async shutdown drain timed out with queued events still pending."
" Remaining events may be discarded."
);
```

If queue count or queued byte size is already available at that point, including those values in the message would be even better. But the main request is the warning itself, not a specific implementation.

## Relevant code

- `agent/native/ext/backend_comm.cpp`
- function `backgroundBackendCommThreadFunc_shouldBreakLoop`

## Environment

- Ecentria fork: [ecentria/apm-agent-php](https://github.com/ecentria/apm-agent-php)
- PHP 7.3
- Elastic APM PHP agent 1.15.1
- CLI SAPI
- async backend communication enabled

## Additional context

- This request is about observability of the shutdown path, not necessarily changing shutdown behavior.
- If useful, I can provide the CLI workload pattern and agent logs from the failing run.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.