cloudfoundry / cloudfoundry/loggregator-agent-release
Message retry performance implications and architectural issues
Open
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 14
- Forks
- 34
- Avg merge
- 7h 6m
- Merged PRs (30d)
- 2
Description
This is an issue to discuss the current state of the retry logic for syslog messages,
As there are some implications, that are problematic.
just listed here shortly for an overview:
- having a syslog drain fail with high load will drop messages for other drains.
This will also put the cpu consumption of the syslog agent over 1 cpu, not sure why - the syslog-batching implementation is not able to use the retry mechanic, as there is no state about the batching being done in the retry writer.
I will add details and my testing results here later in a better formatted way.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the syslog message retry logic, the syslog-batching implementation, and the retry writer. Reproduce the high-load drain failure and CPU behavior, then document the batching and retry-state requirements needed to prevent message drops across other drains.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100