Graylog2 / Graylog2/graylog2-server
Collector FAILED remote config status causes infinite re-offer loop
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Current behavior
`OpAmpService#handleRemoteConfig` only advances the per-instance `lastProcessedTxnSeq` transaction cursor on `RemoteConfigStatuses_APPLIED`. A `FAILED` status is logged at DEBUG, but the error message is not persisted.
Because the cursor doesn't move, the config-change transactions remain "unprocessed" and the server re-offers the same config in the response to every subsequent `AgentToServer` message. The Collector rolls back, retries the re-offered config, fails again, and reports FAILED again: an indefinite restart/rollback loop and no operator visibility.
## Possible Fix
- On `FAILED`, parse `last_remote_config_hash` and advance the `lastProcessedTxnSeq`, same as the APPLIED branch.
The broken config is no longer re-offered. The Collector keeps running the previous working config until
the config changes again.
- Advance monotonically (`{"$max": {"last_processed_txn_seq": 23}}`) and only *to* the failed seq, so
newer pending transactions are still offered on the next message.
- Track applied state separately: keep a new `lastAppliedTxnSeq` (APPLIED only) alongside
the `lastProcessedTxnSeq`, so we can tell what a collector is actually running.
- Persist the failure on the instance document and clear it on the next APPLIED. (see below)
- Show the failure in the UI (instance detail and fleet-level aggregation, e.g.
"3 of 120 collectors failed to apply seq 42").
- Include failures in the new health system.
```json
{
"last_processed_txn_seq": "23",
"last_applied_txn_seq": "22",
"last_config_failure": {
"txn_seq": "23",
"error_message": "...",
"failed_at": "2026-08-06T09:14:07.298Z"
}
}
```
Contributor guide
Research direction
Start at OpAmpService#handleRemoteConfig and trace how APPLIED and FAILED statuses update the per-instance transaction cursor. The issue lists several possible fixes spanning failure persistence, UI aggregation, and health reporting, but does not identify files or tests or select one scope; completion criteria need to be agreed before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100