fluent / fluent/fluent-bit

Log to Metrics generates histogram buckets with invalid scientific notation for large values

Open
#11,651 0 comments 1 reaction 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

**Describe the bug**

When using the `log_to_metrics` filter to generate Prometheus histograms, bucket boundaries with values greater than or equal to 1,000,000 are emitted in scientific notation (e.g. `1e+06`) but the format is not correct since it adds a decimal to this scientific notation. The error that you get in prometheus when you do a `histogram_quantile` is:

```
PromQL warning: bucket label "le" is missing or has a malformed value of "1e+08.0" for metric name "" (1:26)
```

As you can see the metrics that those buckets generate:

```bash
apache_http_histogram_request_size_bytes_bucket{le="100.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 0
apache_http_histogram_request_size_bytes_bucket{le="1000.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3905
apache_http_histogram_request_size_bytes_bucket{le="10000.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="100000.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
# these ones are not processed correctly
apache_http_histogram_request_size_bytes_bucket{le="1e+06.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="1e+07.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="1e+08.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
apache_http_histogram_request_size_bytes_bucket{le="1e+09.0",_cluster="playground1",_bundle="dev1",_host="XXX",proxy_type="ingest"} 3959
```

**To Reproduce**
- Configure a `log_to_metrics` filters with these buckets:
```
[FILTER]
Name log_to_metrics
Match httpd_access*
Tag metrics_httpd_access
Metric_Mode histogram
Metric_Name request_size_bytes
Metric_Description This metric shows the request size in bytes
Metric_Namespace apache_http
Value_Field requestsize
Label_Field _cluster
Label_Field _bundle
Label_Field _host
Label_Field proxy_type
Bucket 100
Bucket 1000
Bucket 10000
Bucket 100000
Bucket 1000000
Bucket 10000000
Bucket 100000000
Bucket 1000000000
Kubernetes_Mode off
Discard_Logs off
Flush_Interval_Sec 10
Flush_Interval_Nsec 0
Emitter_Mem_Buf_Limit 10M

[OUTPUT]
Name prometheus_exporter
Match metrics_httpd_access
Host 127.0.0.1
Port 9999
```
- Curl to `localhost:9999` and you can see the wrong scientific notation in prometheus

**Expected behavior**
- I think the scientific notation is accepted, but not with the decimal since it's wrong scientific notation.

**Screenshots**

**Your Environment**

* Version used: 4.1.0
* Configuration:
* Environment name and version (e.g. Kubernetes? What version?):
* Server type and version:
* Operating System and version: Alma9
* Filters and plugins: log_to_metrics

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.