fluent / fluent/fluent-bit

Allow fluentbit exec plugin to process forevergoing commands like mpstat

Open
#6,518 12 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

**Is your feature request related to a problem? Please describe.**
I would like to be able to parse outputs of these commands with exec plugin:
iostat -d 1
mpstat 1
pidstat 1
These are neverending commands but fluentbit expects the command to finish. However for example fluentd manages this and creates record from every line these commands output at the time they output it - continually.

**Describe the solution you'd like**
In fluentd when you dont include the alternative of fluentbits "Interval_Sec" it runs once and it does not matter if the command finishes or not, it processes all the lines the command outputs continually. In fluentbit there is already the "Oneshot" parametr. I am not sure about its behaviour, in documentations it says: "This allows collection of data precedent to fluent-bit's startup (bool, default: false)" - I am dont get how could I collect data precedent to fluentd-bits startup with that. I guess that if you use Oneshot parametr you dont have to use Interval_Sec parametr and command goes once? Since this is unclear for me, its hard for me to suggest solution. The best would be if the fluentbit could process the lines that the command outputs, no matter if the command finishes or not or what parameters are set, out of the box.

**Describe alternatives you've considered**
- None came to my mind, maybe there are some, but continuous output commands exist and if not for these commands the need might arise for some other command.

**Additional context**
I am trying be flexible in what I monitor and exec almost allows that. Also using exec is easy because you can use any bash command without the need to learn how to do go custom plugins.

I was trying this config and it outputs only after I kill the fluentbit process (you need to have sysstat package installed, its comonly available in apt, dnf,...)

path/to/fluent-bit -c fluent-bit.conf

```
[SERVICE]
Flush 5
Daemon Off
Log_Level info
parsers_file /home/vagrant/fluentbit_parsers.conf
storage.path /home/vagrant/storage

[INPUT]
Name exec
tag psql_statements
command pidstat -dHhurv 1 | grep --line-buffered -E '^[0-9]' | stdbuf -oL -eL tr -s " \t" " "
Oneshot true
storage.type filesystem
Buf_Size 500mb
mem_buf_limit 30mb

[OUTPUT]
Name stdout
Match *

```

Contributor guide

Open the contributing guide

Research direction

Start with the exec input plugin and reproduce the provided pidstat configuration, including Oneshot behavior and the command pipeline. Define done as processing each line from a never-ending command as it is emitted, rather than waiting until fluent-bit exits, while clarifying how Interval_Sec and Oneshot should interact.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.