fluent / fluent/fluentd

enable_input_metrics: inaccurate values

Open
#4,717 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Describe the bug

We can take metrics of input plugins by setting [enable_input_metrics](https://docs.fluentd.org/deployment/system-config#enable_input_metrics).
However, the value would be inaccurate.

I don't directly confirm it yet, but I have confirmed `@caller_plugin_id` of `EventRouter` has race condition.

* https://github.com/fluent/fluentd/issues/4567#issuecomment-2503380410

So, it would be possible that a wrong `metric_callbacks` is selected.

https://github.com/fluent/fluentd/blob/2d8c9d4b94b1d45e7d4d6e59caae5640a516eabd/lib/fluent/event_router.rb#L96-L102

It would cause wrong metrics calculation.

### To Reproduce

I haven't checked it yet, but the following settings should result in a slight error in the metric values.

```xml

enable_input_metrics

@type monitor_agent

@type sample
tag test.foo
rate 100

@type sample
tag test.bar
rate 100

@type sample
tag test.boo
rate 100

@type null

```

Wait a few minutes and check the metrics.

```bash
curl http://localhost:24220/api/plugins.json | jq
```

```json
{
"plugins": [
{
"plugin_id": "object:d34",
"plugin_category": "input",
"type": "monitor_agent",
"config": {
"@type": "monitor_agent"
},
"output_plugin": false,
"retry_count": null,
"emit_records": 0,
"emit_size": 0
},
{
"plugin_id": "object:d48",
"plugin_category": "input",
"type": "sample",
"config": {
"@type": "sample",
"tag": "test.foo",
"rate": "100"
},
"output_plugin": false,
"retry_count": null,
"emit_records": 43112,
"emit_size": 0
},
{
"plugin_id": "object:d5c",
"plugin_category": "input",
"type": "sample",
"config": {
"@type": "sample",
"tag": "test.bar",
"rate": "100"
},
"output_plugin": false,
"retry_count": null,
"emit_records": 43109,
"emit_size": 0
},
{
"plugin_id": "object:d70",
"plugin_category": "input",
"type": "sample",
"config": {
"@type": "sample",
"tag": "test.boo",
"rate": "100"
},
"output_plugin": false,
"retry_count": null,
"emit_records": 43109,
"emit_size": 0
},
{
"plugin_id": "object:d0c",
"plugin_category": "output",
"type": "null",
"config": {
"@type": "null"
},
"output_plugin": true,
"retry_count": 0,
"emit_records": 129330,
"emit_size": 0,
"emit_count": 129330,
"write_count": 0,
"rollback_count": 0,
"slow_flush_count": 0,
"flush_time_count": 0,
"retry": {}
}
]
}
```

You can confirm a difference in the value of `emit_records` for each `in_sample`.

### Expected behavior

There is no difference in the value of `emit_records` for each `in_sample`.

### Your Environment

```markdown
- Fluentd version: 1.18.0
- Package version:
- Operating system: Ubuntu 20.04.6 LTS (Focal Fossa)
- Kernel version: 5.15.0-124-generic
```

### Your Configuration

```apache
Noted in `To Reproduce`.
```

### Your Error Log

```shell
No error.
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with lib/fluent/event_router.rb at lines 96-102 and trace how @caller_plugin_id selects metric_callbacks when enable_input_metrics is enabled. Reproduce the configuration using three sample inputs, monitor_agent, and the plugins API, then compare emit_records across inputs. Done means the input metrics remain accurate and the race condition has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.