fluent / fluent/fluent-bit

Default Workers causes SIGSEGV

Open
#7,552 10 comments 0 reactions 1 assignee Claimed by @braydonk View on GitHub
exempt-stale
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

Hi,
I use fluent-bit deployed with helm chart (latest chart fluent-bit-0.30.3 + app version 2.1.4) and I noticed that my pod is constantly restarting. It works almost fine, logs are sent but part of logs are missing and there are like 20 pod restarts per day. I change default (2) value of Workers to 1 and seems it works correctly now.
I use stackdriver output plugin.

conf below:
```
data:
custom_parsers.conf: |
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
fluent-bit.conf: |
[SERVICE]
Daemon Off
Flush 1
Log_Level info
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On

[INPUT]
Name tail
Path /var/log/containers/test_container1*.log, /var/log/containers/test_container2*.log, /var/log/containers/test_container3-*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On

[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
tls.verify Off
K8S-Logging.Parser On
K8S-Logging.Exclude On

[OUTPUT]
Name stackdriver
Match kube.*
google_service_credentials /etc/config/gcp.json
resource k8s_container
net.dns.resolver LEGACY
tls On
tls.verify Off
tls.crt_file /etc/config/cert.crt
tls.key_file /etc/config/key.pem
tls.debug 4
severity_key level
k8s_cluster_name GUM
k8s_cluster_location gum-server
Workers 1
```
error traces: (the editor replaced enters with spaces, don't know why)

1
```
stderr F [2023/06/06 11:37:32] [engine] caught signal (SIGSEGV)
stderr F #0 0x7fe679edaf87 in ???() at ???:0
stderr F #1 0x7fe679ed9402 in ???() at ???:0
stderr F #2 0x7fe679ed94e1 in ???() at ???:0
stderr F #3 0x7fe679ed9454 in ???() at ???:0
stderr F #4 0x7fe67a03ecdb in ???() at ???:0
stderr F #5 0x7fe679ed928d in ???() at ???:0
stderr F #6 0x7fe679ed9454 in ???() at ???:0
stderr F #7 0x7fe67a01e60f in ???() at ???:0
stderr F #8 0x7fe67a1585be in ???() at ???:0
stderr F #9 0x55a0f452d878 in tls_session_destroy() at src/tls/openssl.c:391
stderr F #10 0x55a0f452ed60 in flb_tls_session_destroy() at src/tls/flb_tls.c:651
stderr F #11 0x55a0f44d3685 in prepare_destroy_conn() at src/flb_upstream.c:446
stderr F #12 0x55a0f44d43cd in flb_upstream_conn_timeouts() at src/flb_upstream.c:862
stderr F #13 0x55a0f449707d in cb_thread_sched_timer() at src/flb_output_thread.c:58
stderr F #14 0x55a0f44c4611 in flb_sched_event_handler() at src/flb_scheduler.c:428
stderr F #15 0x55a0f44977e5 in output_thread() at src/flb_output_thread.c:266
stderr F #16 0x55a0f44d861f in step_callback() at src/flb_worker.c:43
stderr F #17 0x7fe67a3f2ea6 in ???() at ???:0
stderr F #18 0x7fe679ca5a2e in ???() at ???:0
stderr F #19 0xffffffffffffffff in ???() at ???:0
```

2
```
stderr F [2023/06/07 08:15:30] [ info] [output:stackdriver:stackdriver.0] Cached token is expired. Waiting on lock.
stderr F [2023/06/07 08:15:30] [engine] caught signal (SIGSEGV)
stderr F [2023/06/07 08:15:30] [engine] caught signal (SIGSEGV)
stderr F #0 0x56495e309767 in __mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:142
stderr F #1 0x56495e309792 in mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:147
stderr F #2 0x56495e3099f4 in flb_kv_item_destroy() at src/flb_kv.c:89
stderr F #3 0x56495e309a49 in flb_kv_release() at src/flb_kv.c:101
stderr F #4 0x56495e3279a5 in http_headers_destroy() at src/flb_http_client.c:1027
stderr F #5 0x56495e3284f7 in flb_http_client_destroy() at src/flb_http_client.c:1394
stderr F #6 0x56495e629776 in cb_stackdriver_flush() at plugins/out_stackdriver/stackdriver.c:2624
stderr F #7 0x56495e25214d in output_pre_cb_flush() at include/fluent-bit/flb_output.h:559
stderr F #8 0x56495ed5dfc6 in co_init() at lib/monkey/deps/flb_libco/amd64.c:117
stderr F #9 0xffffffffffffffff in ???() at ???:0
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.