fluent / fluent/fluent-bit

Additional metrics needed

Open
#2,968 12 comments 4 reactions 1 assignee Claimed by @edsiper View on GitHub
enhancement fixed
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

**Is your feature request related to a problem? Please describe.**
We would like to have the following metrics exposed in Prometheus format:
1. Total number of records that were attempted to be written as output, including both those that succeeded as well as those that failed. This is different than `fluentbit_output_proc_records_total` which only count those that were [successfully written](https://github.com/fluent/fluent-bit/blob/896c55b33fc27fd39a20e941c1f7be8d996a7066/src/flb_output.c#L841).

2. Total number of records that were retried. The count that is available right now shows only the number of [actual retries](https://github.com/fluent/fluent-bit/blob/896c55b33fc27fd39a20e941c1f7be8d996a7066/src/flb_output.c#L841).

3. API Request count - the number of times that the api was called to write the logs. Currently, there is only a record of the number of requests that failed (fluentbit_output_retries_total) but not the number of successful API calls, only records.

4. Uptime. Currently, the uptime is only available as a REST API, and is not available via Prometheus.

Finally, for the successful and failed entry counts and api calls above, it would be an added benefit to add a label with the HTTP response code.

This will allow easier mapping to the [metrics collected by GCP for the agents](http://cloud/monitoring/api/metrics_agent).

**Describe the solution you'd like**
It should be possible to add a few extra counter to [handle_output_event](https://github.com/fluent/fluent-bit/blob/1913e337b61dcd8c5ef78817a6a977675412dfde/src/flb_engine.c#L121) to keep track of full number of api calls and full number of entries that the system attempted to write.

**Describe alternatives you've considered**
If this is not possible, we could attempt to add it in to the Stackdriver plugin, but this would not make it useful to other systems.

**Additional context**
The purpose is to more easily be able to map existing FluentBit metrics exposed via Prometheus to the [metrics expected by Stackdriver for its agents](http://cloud/monitoring/api/metrics_agent).

Specifically, we would need accurate metrics for:
* log_entry_count
* log_entry_retry_count
* request_count
* uptime

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.