Windows: endless "No error" log until disk is full
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
**Describe the bug**
All of a sudden, Fluentbit starts logging the following errors, until it fills up the entire disk of the server.
This seems related to https://github.com/fluent/fluent-bit/issues/3146 , as the actual error is not reported, and it's pretty hard to find out the root cause.
This only happens on Windows. We're running v2.0.6 on Kubernetes (AKS v1.24.6)
```
2022-12-14T02:27:26.6296592Z stderr F write: No error
2022-12-14T02:27:26.6302562Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6302979Z stderr F write: No error
2022-12-14T02:27:26.6308588Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6309359Z stderr F write: No error
2022-12-14T02:27:26.6312701Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6312701Z stderr F write: No error
2022-12-14T02:27:26.631804Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6318579Z stderr F write: No error
2022-12-14T02:27:26.6324806Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6324806Z stderr F write: No error
2022-12-14T02:27:26.6324806Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6324806Z stderr F write: No error
2022-12-14T02:27:26.633578Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6336124Z stderr F write: No error
2022-12-14T02:27:26.6338881Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6338881Z stderr F write: No error
2022-12-14T02:27:26.6346284Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
2022-12-14T02:27:26.6346629Z stderr F write: No error
2022-12-14T02:27:26.6350891Z stderr F [2022/12/14 02:27:26] [error] [C:\src\src\flb_utils.c:368 errno=0] No error
```
**To Reproduce**
It happens randomly, with no way to reproduce it. However, the missing real error number should be fairly easy to reproduce. See https://github.com/fluent/fluent-bit/issues/3146
**Expected behavior**
- Log the actual error, not a "No error", which is unhelpful and misleading
- Implement a backoff to avoid logging continuosly until some resources are exhausted
**Screenshots**
**Your Environment**
* Version used: v2.0.6
* Configuration:
* Environment name and version (e.g. Kubernetes? What version?): Kubernetes (AKS v1.24.6)
* Server type and version: N/A
* Operating System and version: Microsoft Windows Server 2022 Datacenter (10.0.20348.0)
* Filters and plugins:
Config
```
[SERVICE]
Flush 5s
# Daemon
# ======
# Instruct Fluent Bit to run in foreground or background mode.
Daemon Off
Log_Level warn
# Parsers_File
# ============
# Specify an optional 'Parsers' configuration file
Parsers_File parsers.conf
Plugins_File plugins.conf
# HTTP Server
# ===========
# Enable/Disable the built-in HTTP Server for metrics
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
[INPUT]
Name winlog
Channels System,Security,Application
Tag host.event.*
Interval_Sec 1
DB winlog.sqlite
[INPUT]
# I0122 18:02:48.137905 5440 reconciler.go:301] Volume detached for volume "k8s-collection-fluent-bit-token-9k2pb" (UniqueName: "kubernetes.io/secret/6d90e64c-3cbe-11ea-ab74-a25d3bc6e566-k8s-collection-fluent-bit-token-9k2pb") on node "akswin000000" DevicePath ""
# 2020/01/22 18:02:46 [3292] [ipam] Refreshing address source.
Name tail
Path C:\k\kubelet.err.log
Parser raw
Path_Key kubelet_err_log
DB C:\tail2.db
Tag host.kubelet.*
Refresh_Interval 60
Rotate_Wait 5
#Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB.Sync Normal
Buffer_Chunk_Size 2MB
Buffer_Max_Size 10MB
[INPUT]
Name tail
Path C:\CalicoWindows\logs\calico-felix.log
Parser raw
Path_Key felix_log
DB C:\tail4.db
Tag host.calico.felix.*
Refresh_Interval 60
Rotate_Wait 5
#Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB.Sync Normal
Buffer_Chunk_Size 2MB
Buffer_Max_Size 10MB
[INPUT]
Name tail
Path C:\CalicoWindows\logs\calico-felix.err.log
Parser raw
Path_Key felix_err_log
DB C:\tail4.db
Tag host.calico.felix.*
Refresh_Interval 60
Rotate_Wait 5
#Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB.Sync Normal
Buffer_Chunk_Size 2MB
Buffer_Max_Size 10MB
[INPUT]
Name tail
Path C:\CalicoWindows\logs\calico-node.log
Parser raw
Path_Key calico_node_log
DB C:\tail5.db
Tag host.calico.node.*
Refresh_Interval 60
Rotate_Wait 5
#Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB.Sync Normal
Buffer_Chunk_Size 2MB
Buffer_Max_Size 10MB
[INPUT]
Name tail
Path C:\CalicoWindows\logs\calico-node.err.log
Parser raw
Path_Key calico_node_err_log
DB C:\tail5.db
Tag host.calico.node.*
Refresh_Interval 60
Rotate_Wait 5
#Mem_Buf_Limit 5MB
Skip_Long_Lines On
DB.Sync Normal
Buffer_Chunk_Size 2MB
Buffer_Max_Size 10MB
[FILTER]
Name modify
Match host.*
Add _HOSTNAME ${NODE_NAME}
[FILTER]
Name kubernetes
Match containers.*
# We have to use the ip since kubernetes.default.svc is not resolvable from a windows container
# NOTE we need to add a sleep at the startup of the container to be sure that the windows network is ready for the container
#Kube_URL https://kubernetes.default.svc:443
# required for windows https://github.com/fluent/fluent-bit/issues/5381
tls.verify Off
Kube_URL https://kubernetes.default.svc.cluster.local:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
[OUTPUT]
Name azure
Match *
Customer_ID [REDACTED]
Shared_Key [REDACTED]
Retry_Limit 5
# required for windows https://github.com/fluent/fluent-bit/issues/5381
tls.verify Off
net.dns.mode TCP
```
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.