fluent / fluent/fluent-bit

Windows: endless "No error" log until disk is full

Open
#6,626 6 comments 3 reactions 0 assignees View on GitHub
exempt-stale status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

**Describe the bug**

All of a sudden, Fluentbit starts logging the following errors, until it fills up the entire disk of the server.
This seems related to https://github.com/fluent/fluent-bit/issues/3146 , as the actual error is not reported, and it's pretty hard to find out the root cause.
This only happens on Windows. We're running v2.0.6 on Kubernetes (AKS v1.24.6)

```
2022-12-14T02:27:26.6296592Z stderr F write: No error
2022-12-14T02:27:26.6302562Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6302979Z stderr F write: No error
2022-12-14T02:27:26.6308588Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6309359Z stderr F write: No error
2022-12-14T02:27:26.6312701Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6312701Z stderr F write: No error
2022-12-14T02:27:26.631804Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6318579Z stderr F write: No error
2022-12-14T02:27:26.6324806Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6324806Z stderr F write: No error
2022-12-14T02:27:26.6324806Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6324806Z stderr F write: No error
2022-12-14T02:27:26.633578Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6336124Z stderr F write: No error
2022-12-14T02:27:26.6338881Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6338881Z stderr F write: No error
2022-12-14T02:27:26.6346284Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6346629Z stderr F write: No error
2022-12-14T02:27:26.6350891Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
```

**To Reproduce**
It happens randomly, with no way to reproduce it. However, the missing real error number should be fairly easy to reproduce. See https://github.com/fluent/fluent-bit/issues/3146

**Expected behavior**

- Log the actual error, not a "No error", which is unhelpful and misleading
- Implement a backoff to avoid logging continuosly until some resources are exhausted

**Screenshots**

**Your Environment**

* Version used: v2.0.6
* Configuration:
* Environment name and version (e.g. Kubernetes? What version?): Kubernetes (AKS v1.24.6)
* Server type and version: N/A
* Operating System and version: Microsoft Windows Server 2022 Datacenter (10.0.20348.0)
* Filters and plugins:

Config

```
[SERVICE]
Flush 5s

# Daemon
# ======
# Instruct Fluent Bit to run in foreground or background mode.
Daemon Off
Log_Level warn

# Parsers_File
# ============
# Specify an optional 'Parsers' configuration file
Parsers_File parsers.conf
Plugins_File plugins.conf

# HTTP Server
# ===========
# Enable/Disable the built-in HTTP Server for metrics
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020

[INPUT]
Name winlog
Channels System,Security,Application
Tag host.event.*
Interval_Sec 1
DB winlog.sqlite

[INPUT]
# I0122 18:02:48.137905 5440 reconciler.go:301] Volume detached for volume "k8s-collection-fluent-bit-token-9k2pb" (UniqueName: "kubernetes.io/secret/6d90e64c-3cbe-11ea-ab74-a25d3bc6e566-k8s-collection-fluent-bit-token-9k2pb") on node "akswin000000" DevicePath ""
# 2020/01/22 18:02:46 [3292] [ipam] Refreshing address source.
Name tail
Path C:\k\kubelet.err.log
Parser raw
Path_Key kubelet_err_log
DB C:\tail2.db
Tag host.kubelet.*
Refresh_Interval 60
Rotate_Wait 5
#Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB.Sync Normal
Buffer_Chunk_Size 2MB
Buffer_Max_Size 10MB

[INPUT]
Name tail
Path C:\CalicoWindows\logs\calico-felix.log
Parser raw
Path_Key felix_log
DB C:\tail4.db
Tag host.calico.felix.*
Refresh_Interval 60
Rotate_Wait 5
#Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB.Sync Normal
Buffer_Chunk_Size 2MB
Buffer_Max_Size 10MB
[INPUT]
Name tail
Path C:\CalicoWindows\logs\calico-felix.err.log
Parser raw
Path_Key felix_err_log
DB C:\tail4.db
Tag host.calico.felix.*
Refresh_Interval 60
Rotate_Wait 5
#Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB.Sync Normal
Buffer_Chunk_Size 2MB
Buffer_Max_Size 10MB

[INPUT]
Name tail
Path C:\CalicoWindows\logs\calico-node.log
Parser raw
Path_Key calico_node_log
DB C:\tail5.db
Tag host.calico.node.*
Refresh_Interval 60
Rotate_Wait 5
#Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB.Sync Normal
Buffer_Chunk_Size 2MB
Buffer_Max_Size 10MB
[INPUT]
Name tail
Path C:\CalicoWindows\logs\calico-node.err.log
Parser raw
Path_Key calico_node_err_log
DB C:\tail5.db
Tag host.calico.node.*
Refresh_Interval 60
Rotate_Wait 5
#Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB.Sync Normal
Buffer_Chunk_Size 2MB
Buffer_Max_Size 10MB

[FILTER]
Name modify
Match host.*
Add _HOSTNAME ${NODE_NAME}

[FILTER]
Name kubernetes
Match containers.*
# We have to use the ip since kubernetes.default.svc is not resolvable from a windows container
# NOTE we need to add a sleep at the startup of the container to be sure that the windows network is ready for the container
#Kube_URL https://kubernetes.default.svc:443
# required for windows https://github.com/fluent/fluent-bit/issues/5381
tls.verify Off
Kube_URL https://kubernetes.default.svc.cluster.local:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed

[OUTPUT]
Name azure
Match *
Customer_ID [REDACTED]
Shared_Key [REDACTED]
Retry_Limit 5
# required for windows https://github.com/fluent/fluent-bit/issues/5381
tls.verify Off
net.dns.mode TCP
```

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.