elastic / elastic/elastic-agent
Add support for ingesting files from a shared volume across multiple hosts in a HA configuration
- Dominant language
- Go
- Stars
- 276
- Forks
- 266
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 312
Description
The architecture is: HA deployment with 3 servers, one primary and 2 replicas. The data (including the log files Elastic Agent is ingesting) is on a volume replicated on all servers, however it is only mounted on the primary. When there is a fail over event, the volume is unmounted from the primary and mounted onto one of the replicas that becomes the primary.
Each server has its own Elastic Agent, the Elastic Agent mounted on the replica should see the log file and continue ingestion from where the previous primary stopped.
Contributor guide
Research direction
Start by tracing how Elastic Agent stores ingestion state and resumes file processing after a host or primary change. Review the failover scenario described in this issue and determine how state can remain available when the shared volume moves between hosts. Done means the replacement primary continues ingesting each log from the previous primary's position without losing or duplicating data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100