vectordotdev / vectordotdev/vector

Component activity heartbeat / canoncial log lines

Open
#1,246 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

domain: observability meta: idea needs: approval needs: requirements
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

This is a follow up to https://github.com/timberio/vector/pull/1237. Specifically this quote:

I also like enabling blackholes with high print_amount settings anyway, so that my journald logs show some progress occasionally.

I like this idea, but it overlaps somewhat with observing Vector in general. When Vector supplies it's own "internal" metrics source the user could then build a pipeline to visualize that data however they wish. In addition, we could provide a light-weight Vector sub-command on the CLI that could use this data to provide insight into a running Vector instance. That said, this still doesn't entirely cover the use case quoted above, which is basically a "heartbeat" within the logs themselves.

What I'm wondering is if we should consider canoncial log lines for each component that could be printed at a configurable rate. This could include any information it wants, such as the number of successful events processed, number of failures, timing information, and more. But how would this overlap with our own internal metrics, if at all?

Before beginning work we should decide this is worth doing and formulate a spec that we all agree on. This should take into consideration upcoming observability work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading pull request #1237 and the linked Stripe canonical log lines discussion. Before implementation, formulate an agreed specification covering component heartbeat log lines, configurable rates, overlap with internal metrics, and upcoming observability work. Done means the project has decided whether this is worthwhile and documented the agreed scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.