fluent / fluent/fluent-plugin-prometheus
Provide way to remove metric that are not published anymore
- Dominant language
- Ruby
- Stars
- 257
- Forks
- 83
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 7
Description
It seem currently exporter has no way to stop displaying metrics which are obsolete(never going to be updated).
For example in kubernetes cluster we can have thousands of containers recreated every hour.
We are collecting fluentd logging stats ( message rate with flowcount plugin) per each container and use prometheus to aggregate this stats. For example we have "fluentd_flowcounter_count_rate" metric and then we use prometheus labels to tag individual container/pod this metric belong to. This works fine, only problem that fluentd prometheus exporter keep showing metrics which are not published for a long time and obsolete (flowcount does not report this logfile stats anymore, log file removed, container deleted). With our rate of deletion/creation of containers output of the prometheus exporter quickly becomes polluted with large amount of obsolete metrics.
Is there a way to make fluent-plugin-prometheus stop publishing idle metrics?
Maybe it's possible to introduce extra attribute for the metric to specify idle timeout after which metric will be removed from publishing?
**System info**:
fluentd-0.12.34
'fluent-plugin-prometheus' : '0.3.0'
**Fluentd prometheus output configuration**:
```
@type copy
type flowcounter
count_keys *
unit minute
aggregate tag
output_style tagged
delete_idle true
@type record_transformer
enable_ruby true
remove_keys kubernetes_pod_name,kubernetes_namespace,app,job,instance,pod_template_generation,version
fluentd-tag ${record['tag']}
@type prometheus
tag ${fluentd-tag}
name fluentd_flowcounter_count_rate
type gauge
desc count rate
key count_rate
```
**Output of the exporter**:
\# TYPE fluentd_flowcounter_count_rate gauge
\# HELP fluentd_flowcounter_count_rate count rate
fluentd_flowcounter_count_rate{tag="kubernetes.var.log.containers.test_service.4.log"} 0.1
fluentd_flowcounter_count_rate{tag="kubernetes.var.log.containers.test_service.7.log"} 0.05
fluentd_flowcounter_count_rate{tag="kubernetes.var.log.containers.test_service.5.log"} 0.16
fluentd_flowcounter_count_rate{tag="kubernetes.var.log.containers.test_service.1.log"} 0.13
fluentd_flowcounter_count_rate{tag="kubernetes.var.log.containers.test_service.6.log"} 0.08
fluentd_flowcounter_count_rate{tag="kubernetes.var.log.containers.test_service.9.log"} 0.05
fluentd_flowcounter_count_rate{tag="kubernetes.var.log.containers.test_service.2.log"} 0.13
fluentd_flowcounter_count_rate{tag="kubernetes.var.log.containers.test_service.8.log"} 0.15
fluentd_flowcounter_count_rate{tag="kubernetes.var.log.containers.test_service.3.log"} 0.18
fluentd_flowcounter_count_rate{tag="kubernetes.var.log.containers.test_service.10.log"} 0.11
For example after we delete kubernetes.var.log.containers.test_service.10.log and kubernetes.var.log.containers.test_service.9.log they still will be displayed by exporter even they never will be updated anymore.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.