fluent / fluent/fluent-bit

Python multiline parser is not working with systemd input

Open
#9,290 8 comments 0 reactions 0 assignees View on GitHub
exempt-stale status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

My setup is somewhat similar to https://github.com/fluent/fluent-bit/issues/8787

I have several containers running on podman on RHEL8 EC2.

I use fluent bit to forward the container logs to cloudwatch.

I run my container stack using docker compose.

This is the fluent bit service definition.
```
logs:
image: fluent/fluent-bit:3.0.6
restart: always
user: root
network_mode: host
command: /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf
volumes:
- /run/log/journal:/run/log/journal:z
- /etc/machine-id:/etc/machine-id:ro
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:z
- ./parsers.conf:/fluent-bit/etc/parsers.conf:z
```

`fluent-bit.conf`
```
[SERVICE]
Parsers_File /fluent-bit/etc/parsers.conf

[INPUT]
Name systemd
Tag dd-service
Path /run/log/journal
Read_From_Tail On
Systemd_Filter _COMM=conmon
Systemd_Filter CONTAINER_TAG=dd-service
Systemd_Filter_Type And

[INPUT]
Name systemd
Tag yuma
Path /run/log/journal
Read_From_Tail On
Systemd_Filter _COMM=conmon
Systemd_Filter CONTAINER_TAG=yuma
Systemd_Filter_Type And

[INPUT]
Name systemd
Tag db
Path /run/log/journal
Read_From_Tail On
Systemd_Filter _COMM=conmon
Systemd_Filter CONTAINER_TAG=db
Systemd_Filter_Type And

[INPUT]
Name systemd
Tag app
Path /run/log/journal
Read_From_Tail On
Systemd_Filter _COMM=conmon
Systemd_Filter CONTAINER_TAG=app
Systemd_Filter_Type And

[INPUT]
Name systemd
Tag cron
Path /run/log/journal
Read_From_Tail On
Systemd_Filter _COMM=conmon
Systemd_Filter CONTAINER_TAG=cron
Systemd_Filter_Type And

[INPUT]
Name systemd
Tag server
Path /run/log/journal
Read_From_Tail On
Systemd_Filter _COMM=conmon
Systemd_Filter CONTAINER_TAG=server
Systemd_Filter_Type And

[FILTER]
Name multiline
Match_Regex (app|cron|dd-service)
multiline.key_content MESSAGE
Buffer On
multiline.parser python

[FILTER]
Name parser
Match dd-service
Key_Name MESSAGE
Parser dd-service

[FILTER]
Name parser
Match db
Key_Name MESSAGE
Parser db

[FILTER]
Name parser
Match_Regex (app|cron)
Key_Name MESSAGE
Parser app

[FILTER]
Name parser
Match server
Key_Name MESSAGE
Parser nginx

[OUTPUT]
Name cloudwatch_logs
Match *
region us-east-1
log_group_name soak
log_stream_prefix ec2-
```

`parsers.conf`
```
[PARSER]
Name dd-service
Format regex
Regex (?DEBUG|INFO|WARNING|ERROR|CRITICAL)?:? *(?

[PARSER]
Name db
Format regex
Regex (?

[PARSER]
Name app
Format regex
Regex (?DEBUG|INFO|WARNING|ERROR|CRITICAL):(?[\w\.]+):(User: (?.+?), )?(Customer: (?.+?), )?(from (?[\d+\.]+) )?(with body (?.+?) )?(requested (?[A-Z]+))?(?(?!\/).+?)?((?\/[a-z\/-]*)|[^\/]$).*?(?: at (?

[PARSER]
Name nginx
Format regex
Regex ^(?[^ ]*) (?[^ ]*) (?[^ ]*) \[(?

```
In Cloudwatch, the first 3 lines of the traceback are parsed into one message as expected
```
{
"PRIORITY": "3",
"CODE_FILE": "src/ctr_logging.c",
"CODE_LINE": "320",
"CODE_FUNC": "write_journald",
"_TRANSPORT": "journal",
"_UID": "0",
"_GID": "0",
"_COMM": "conmon",
"_EXE": "/usr/bin/conmon",
"_CAP_EFFECTIVE": "1ffffffffff",
"_SELINUX_CONTEXT": "system_u:system_r:container_runtime_t:s0",
"_SYSTEMD_SLICE": "machine.slice",
"_BOOT_ID": "5d379cbb22d94db78503e2ce48ba675b",
"_MACHINE_ID": "593a407f0bf44f8da32824920ad62d25",
"_HOSTNAME": "ip-172-31-92-238.ec2.internal",
"CONTAINER_ID_FULL": "37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
"CONTAINER_ID": "37a3e93ed098",
"CONTAINER_TAG": "app",
"CONTAINER_NAME": "aliro-orchestrator-app",
"SYSLOG_IDENTIFIER": "app",
"_PID": "3879832",
"_CMDLINE": "/usr/bin/conmon --api-version 1 -c 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -u 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata -p /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/pidfile -n aliro-orchestrator-app --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level info --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/oci-log --log-tag app --conmon-pidfile /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg info --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
"_SYSTEMD_CGROUP": "/machine.slice/libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
"_SYSTEMD_UNIT": "libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
"_SYSTEMD_INVOCATION_ID": "724b21586ffe4e008e0ed099352ae284",
"MESSAGE": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py\", line 55, in inner\n response = get_response(request)\n",
"_SOURCE_REALTIME_TIMESTAMP": "1724783541055290"
}
```
but then it breaks into another message. I expect the entire traceback to be one message.
```
{
"PRIORITY": "3",
"CODE_FILE": "src/ctr_logging.c",
"CODE_LINE": "320",
"CODE_FUNC": "write_journald",
"_TRANSPORT": "journal",
"_UID": "0",
"_GID": "0",
"_COMM": "conmon",
"_EXE": "/usr/bin/conmon",
"_CAP_EFFECTIVE": "1ffffffffff",
"_SELINUX_CONTEXT": "system_u:system_r:container_runtime_t:s0",
"_SYSTEMD_SLICE": "machine.slice",
"_BOOT_ID": "5d379cbb22d94db78503e2ce48ba675b",
"_MACHINE_ID": "593a407f0bf44f8da32824920ad62d25",
"_HOSTNAME": "ip-172-31-92-238.ec2.internal",
"CONTAINER_ID_FULL": "37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
"CONTAINER_ID": "37a3e93ed098",
"CONTAINER_TAG": "app",
"CONTAINER_NAME": "aliro-orchestrator-app",
"SYSLOG_IDENTIFIER": "app",
"_PID": "3879832",
"_CMDLINE": "/usr/bin/conmon --api-version 1 -c 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -u 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata -p /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/pidfile -n aliro-orchestrator-app --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level info --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/oci-log --log-tag app --conmon-pidfile /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg info --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
"_SYSTEMD_CGROUP": "/machine.slice/libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
"_SYSTEMD_UNIT": "libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
"_SYSTEMD_INVOCATION_ID": "724b21586ffe4e008e0ed099352ae284",
"MESSAGE": " ^^^^^^^^^^^^^^^^^^^^^\n",
"_SOURCE_REALTIME_TIMESTAMP": "1724783541055302"
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.