elastic / elastic/beats

[Heartbeat] Implementing sharding for running monitors across multiple machines

Open
#40,438 2 comments 0 reactions 0 assignees View on GitHub
Heartbeat Team:obs-ds-hosted-services
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

### Proposal

There will be a single Heartbeat instance which will act as a coordinator that synchronously talks to remote endpoints that has the ability to run synthetics tests and write the results back.

Sharding will be implemented on the Heartbeat layer where given 10 monitors to run on the next scheduled run or across 100s of monitors, Heartbeat based on the number of machines behind a endpoint, Will shard and send the monitor configuration to the remote synthetic service endpoints using Web socket connection. The remote Synthetics runner will be responsible for running the synthetics monitors and writing the results via Web socket connection.

In addition to that, The connection must be encrypted between the Heartbeat coordinator and the Synthetics monitors runner, We should look in to allowing users to configure custom certificates if necessary to establish the connection.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating the Heartbeat coordinator and the remote synthetics runner, then trace their WebSocket communication. Done would require an agreed sharding design, encrypted connections, and a defined approach for user-configured certificates.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, networking, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.