vectordotdev / vectordotdev/vector
Support configurable metrics aggregations with percentile/histograms transformation
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
For aggregation in for. example statsd would be awesome to have in Vector transformation to do aggregations like min, max, stdev, mean, count, sum on a defined pipe of metrics but what is also needed is to have bucketing to get custom percentiles from a defined list to calculate like
percentiles = [50,70,90,95,97,98,99]
Also, histogram support with an example for ms buckets in duration calculation.
histogram = [1,10,100,1000,2000,3000,4000,5000,6000,7000,8000,9000,10000]
this will add a set of tags based on histogram per aggregated window of metrics. We may also have the option to choose if we like to add histogram and/or percentiles data as new fields or new tags list of buckets - Some systems better support fields query language based on some better query on tags at the end-user experience.
This is connected with https://github.com/timberio/vector/issues/676 to have defined aggregation windows per this transformation.
I think would be nice to move aggregations as most do from source more behind with more control from Vector. about which metrics we like to aggregate and which not.
For this example, we can have current statsd input with some improvement on supported types - https://github.com/timberio/vector/issues/2603 and move aggregation into transforming part to be more elastic. This may also apply for metrics from logs and more sources.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the related aggregation-window issue 676 and the StatsD improvements issue 2603, then trace how the current StatsD input and transformation pipeline handle metrics. Done would require an agreed design and implementation for configurable aggregation windows, percentiles, histograms, and output as fields or tags.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, observability, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100