fluent / fluent/fluent-bit

Add `ignore_missing_path` option to Tail input for optional log file globs

Open
#11,964 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

**Is your feature request related to a problem? Please describe.**

I want to use one shared Fluent Bit Tail input configuration across multiple containerized workloads, but not every workload has the same log file paths.

For example, one workload may write:

```text
/var/log/app/*.log
```

while another may also write optional component logs under:

```text
/var/log/components/*/*.log
```

If the shared Tail config includes both paths, workloads that do not create the optional component log directory can emit noisy missing-path or unmatched-glob errors, even though the absence of those files is expected.

**Describe the solution you'd like**

Add an opt-in Tail input option such as:

```yaml
ignore_missing_path: true
```

Example:

```yaml
pipeline:
inputs:
- name: tail
path: /var/log/app/*.log,/var/log/components/*/*.log
tag: app_logs
ignore_missing_path: true
```

When enabled, Fluent Bit should not emit an error if a configured path or glob does not currently match any files. The Tail input should still refresh normally and begin tailing matching files if they are created later, according to existing behavior such as `refresh_interval`.

The default should preserve current behavior, for example:

```yaml
ignore_missing_path: false
```

**Describe alternatives you've considered**

I have considered maintaining separate Fluent Bit configs per workload type, but that creates duplication and makes config drift more likely.

I have also considered creating placeholder directories/files so optional globs always match something, but that feels like a workaround and adds artificial files only to satisfy the Tail input.

Another option is generating the Tail config dynamically at container startup, but that adds startup scripting and makes the effective runtime config differ from the checked-in config.

**Additional context**

This would be useful for containerized and orchestrated environments where a single logging sidecar or shared Fluent Bit image is reused across heterogeneous workloads.

Some log paths are mandatory, while others are optional depending on enabled features, plugins, or workload type. Having an explicit `ignore_missing_path` option would make optional log paths easier to model without duplicating configs or adding placeholder files.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Fluent Bit Tail input implementation and its handling of unmatched paths, then trace how refresh_interval discovers files created later. Define the option's default and verify that optional unmatched globs produce no error while matching files are still tailed; preserve the current error behavior when the option is disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.