devloperdevesh / devloperdevesh/FaultPlane

[Feat/Telemetry]: Build a Vectorized Non-Blocking Prometheus Aggregation Pipeline utilizing sync/atomic Flags

Open
#19 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
2
Forks
1
Avg merge
14m
Merged PRs (30d)
57

Description

Problem

High-frequency logging setups and scraping metrics collectors under issue #6 perform dynamic allocation routines during rapid parallel tool loops. Locking structural counters during parallel scrape windows triggers tracking jitter and minor data leaks inside telemetry nodes.


Proposed Solution

Refactor the metric pipeline tracking layers in internal/telemetry/collector.go to run on top of an independent Vectorized Lock-Free Metrics Collector Layer.

  • Store operational event counters using fixed arrays of primitive 64-bit integer values mapped explicitly to individual telemetry dimensions.
  • Update tracking counters exclusively utilizing low-overhead atomic addition routines (atomic.AddInt64).
  • Stream compiled tracking data segments down to Prometheus text scrapers utilizing optimized multi-buffer vector writers (unix.Writev) to maximize disk I/O throughput.

Alternatives

  • Utilizing standard Prometheus telemetry libraries, which provide convenient abstraction APIs but introduce object allocation loops inside critical hot routing connection pipelines.

Use Case

Maintains flawless system-wide observability statistics visibility under intense stress workloads without introducing a single byte of memory heap allocation overhead.


Additional Notes

Expose these optimized telemetry output channels cleanly alongside standard dashboards definitions mapped within deployments configs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading internal/telemetry/collector.go to understand the existing metric pipeline tracking layers and how deployment configurations expose dashboard definitions. Compare the current behavior with the requested atomic counters, vectorized output, and zero-allocation stress workload goals. Done means the optimized telemetry channels are exposed cleanly and the pipeline meets those performance requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.