Long delay between EA received input and loaded

**APM Server version** (`apm-server version`): 8.14.3

**Description of the problem including expected versus actual behavior**:

In EA managed apm-server, there are observations that there can be delay between log "received input from elastic-agent" and "loaded input config" in the order of days. It implies 2 servers are actually running in the apm-server process during this long period when the old server is stopping. The fact that 2 TBS gc goroutines may cause #14305 , actually makes #14305 also an evidence that 2 servers are running at the same time. Additionally, as only 1 reload can happen at a time, a long reload will stall other input updates.

Edit: as explained below, there are a few parts to this problem:
- Check and fix processor code that are not concurrent-safe (e.g. monitoring metrics reporting in go-docappender), which would be unsafe when EA hot reloads
- Check and fix code that manipulate global variables, which would be unsafe when EA hot reloads

**Provide logs (if relevant)**:

```
"Oct 7, 2024 @ 19:46:30.592" "loaded input config"
"Oct 7, 2024 @ 19:46:30.577" "tail sampler aborted"
"Oct 7, 2024 @ 19:46:30.576" "LSM aggregator stopped"
"Oct 7, 2024 @ 19:46:30.576" "stopping aggregator"
"Oct 7, 2024 @ 19:46:30.576" "Server stopped"
"Sep 30, 2024 @ 23:41:27.228" "Path /intake/v3/rum/events added to request handler"
"Sep 30, 2024 @ 23:41:27.228" "Path /config/v1/rum/agents added to request handler"
"Sep 30, 2024 @ 23:41:27.228" "Path /intake/v2/rum/events added to request handler"
"Sep 30, 2024 @ 23:41:27.228" "RUM endpoints enabled!"
"Sep 30, 2024 @ 23:41:27.228" "Starting apm-server [ceaf482859c86da0ba6de99005ab2fffae7551c6 built 2024-07-08 17:22:03 +0000 UTC]. Hit CTRL-C to stop it."
"Sep 30, 2024 @ 23:41:27.224" "stopping apm-server... waiting maximum of 30s for queues to drain"
"Sep 30, 2024 @ 23:41:27.224" "Listening on: [::]:8200"
"Sep 30, 2024 @ 23:41:27.224" "Stop listening on: 0.0.0.0:8200"
"Sep 30, 2024 @ 23:41:27.222" "received input from elastic-agent"
```

Start by tracing the elastic-agent hot-reload sequence around the logged "received input from elastic-agent" and "loaded input config" events, including the server shutdown and queue-draining steps. Review processor code such as go-docappender and code that manipulates global variables; done means reloads do not stall for days and concurrent old/new servers remain safe.

Written by the indexing model from the issue text.