fluent / fluent/fluent-operator
bug: ClusterMultilineParser not being registered by Fluent Operator
- Dominant language
- Go
- Stars
- 682
- Forks
- 328
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 13
Description
### Context
When deploying a custom multiline parser configuration, the parser is not being recognized or applied by the FluentBit instances managed by the operator. The Fluent Operator is not properly registering custom ClusterMultilineParser resources, causing container to failed to start.
**Expected Behavior**
The ClusterMultilineParser should be:
- Successfully registered by the Fluent Operator
- Available for use in ClusterInput configurations via the multilineParser field
**Actual Behavior**
- The ClusterMultilineParser resource is created but not registered/recognized
- FluentBit instances do not receive the custom parser configuration which cause the container to failed to start
** Code **
```yaml
---
# Custom Multiline Parser for JSON logs
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterMultilineParser
metadata:
name: spark-json-multiline
labels:
app.kubernetes.io/name: fluent-bit
app.kubernetes.io/component: fluent-bit-config
spec:
type: regex
flushTimeout: 2000
rules:
- start: '/^[IWEF]\d{4} \d{2}:\d{2}:\d{2}\.\d+/'
regex: '^[IWEF]\d{4} \d{2}:\d{2}:\d{2}\.\d+'
next: cont_state
- start: '/^(?![IWEF]\d{4})/'
regex: '^(?![IWEF]\d{4})'
next: cont_state
---
# Updated ClusterInput with custom multiline parser
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
name: fluent-bit-spark-operator
labels:
app.kubernetes.io/name: fluent-bit
app.kubernetes.io/component: fluent-bit-config
spec:
tail:
tag: kube.*
path: /var/log/containers/*-spark-operator-*.log
parser: cri
multilineParser: spark-json-multiline
refreshIntervalSeconds: 10
bufferChunkSize: 32k
bufferMaxSize: 64k
memBufLimit: 5MB
skipLongLines: false
readFromHead: false
db: /fluent-bit/tail/spark-operator-pos.db
dbSync: Normal
---
# ClusterFilter for multiline processing
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFilter
metadata:
name: multiline-filter
labels:
app.kubernetes.io/name: fluent-bit
app.kubernetes.io/component: fluent-bit-config
spec:
match: kube.*
filters:
- multiline:
keyContent: log
buffer: true
emitterMemBufLimit: 120
emitterType: memory
flushMs: 2000
parser: go,cri,spark-json-multiline
---
# ClusterFluentBitConfig to tie everything together
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
name: fluent-bit-config
labels:
app.kubernetes.io/name: fluent-bit
app.kubernetes.io/component: fluent-bit-config
spec:
namespace: kube-system
service:
parsersFile: "parsers.conf"
inputSelector:
matchLabels:
app.kubernetes.io/component: fluent-bit-config
filterSelector:
matchLabels:
app.kubernetes.io/component: fluent-bit-config
multilineParserSelector:
matchLabels:
app.kubernetes.io/component: fluent-bit-config
outputSelector:
matchLabels:
app.kubernetes.io/component: fluent-bit-config
```
**Log**
```
[2025/06/06 05:34:32] [ info] [fluent bit] version=2.2.2, commit=eeea396e88, pid=14 │
│ [2025/06/06 05:34:32] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128 │
│ [2025/06/06 05:34:32] [ info] [cmetrics] version=0.6.6 │
│ [2025/06/06 05:34:32] [ info] [ctraces ] version=0.4.0 │
│ [2025/06/06 05:34:32] [ info] [input:tail:tail.0] initializing │
│ [2025/06/06 05:34:32] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only) │
│ [2025/06/06 05:34:32] [error] [multiline] parser 'spark-json-multiline' not registered │
│ [2025/06/06 05:34:32] [error] [input:tail:tail.0] could not load multiline parsers │
│ [2025/06/06 05:34:32] [error] failed initialize input tail.0 │
│ [2025/06/06 05:34:32] [error] [engine] input initialization failed │
│ [2025/06/06 05:34:32] [error] [lib] backend failed │
│ level=error time=2025-06-06T05:34:33Z msg="Failure during the run time of fluent-bit" error="failed to run fluent-bit: exit status 255"
```
### How did you install fluent operator?
Fluent Operator Version: 2.9.0
Kubernetes Version: v1.30.11
FluentBit Version: 2.2.2
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.