elastic / elastic/elastic-agent

Add support for ingesting files from a shared volume across multiple hosts in a HA configuration

Open
#16,545 1 comment 0 reactions 0 assignees View on GitHub
Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
276
Forks
266
Avg merge
1d 23h
Merged PRs (30d)
312

Description

The architecture is: HA deployment with 3 servers, one primary and 2 replicas. The data (including the log files Elastic Agent is ingesting) is on a volume replicated on all servers, however it is only mounted on the primary. When there is a fail over event, the volume is unmounted from the primary and mounted onto one of the replicas that becomes the primary.

Each server has its own Elastic Agent, the Elastic Agent mounted on the replica should see the log file and continue ingestion from where the previous primary stopped.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how Elastic Agent stores ingestion state and resumes file processing after a host or primary change. Review the failover scenario described in this issue and determine how state can remain available when the shared volume moves between hosts. Done means the replacement primary continues ingesting each log from the previous primary's position without losing or duplicating data.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.