fluent / fluent/fluentd-kubernetes-daemonset
v1.2-debian-cloudwatch crashes where v0.12 works (k8s 1.8.13 CoreOS 17455.0)
- Dominant language
- Ruby
- Stars
- 1.3k
- Forks
- 969
- Avg merge
- 22h 57m
- Merged PRs (30d)
- 1
Description
Deploying for Kubernetes 1.8.13 on CoreOS 1745.5.0 using `fluent/fluentd-kubernetes-daemonset`
Deploying with `v0.12-debian-cloudwatch` works great as in the past, however switching to `v1.2-debian-cloudwatch` and every Pod on every node crash after ~1 minute of run time. Occasionally they get to create a log-flow and even log some entries first, but they always crash. The kept getting restarted but they just crash again. They keep in time too, so after a while they all have exactly e.g. 12 crashes, so I am guess they run the same amount of time before crashing.
Everything else about the config remains unchanged. I wondered if Debian needed more memory so I removed that limit, but an every node in the cluster the container would still run for maybe a minute and then crash.
```
2018-06-15 21:38:39 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2018-06-15 21:38:46 +0000 [info]: using configuration file:
@type null
@type tail
path "/var/log/containers/*.log"
pos_file "/var/log/fluentd-containers.log.pos"
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag "kubernetes.*"
format json
read_from_head true
time_format %Y-%m-%dT%H:%M:%S.%NZ
@type json
time_type string
@type kubernetes_metadata
@type record_transformer
enable_ruby true
kubehost ${record.fetch("kubernetes", Hash.new).fetch("host", "unknown_host")}
@type cloudwatch_logs
log_group_name "anthill-cluster-containers"
log_stream_name_key "kubehost"
remove_log_group_name_key true
auto_create_stream true
put_log_events_retry_limit 20
2018-06-15 21:38:46 +0000 [info]: starting fluentd-1.2.2 pid=5 ruby="2.3.3"
2018-06-15 21:38:46 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby2.3", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"]
2018-06-15 21:38:50 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.5.0'
2018-06-15 21:38:50 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.1.2'
2018-06-15 21:38:50 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.1'
2018-06-15 21:38:50 +0000 [info]: gem 'fluentd' version '1.2.2'
2018-06-15 21:38:50 +0000 [info]: adding match pattern="fluent.**" type="null"
2018-06-15 21:38:50 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2018-06-15 21:38:54 +0000 [info]: adding filter pattern="kubernetes.**" type="record_transformer"
2018-06-15 21:38:54 +0000 [info]: adding match pattern="kubernetes.**" type="cloudwatch_logs"
2018-06-15 21:38:57 +0000 [info]: adding source type="tail"
2018-06-15 21:38:57 +0000 [info]: #0 starting fluentd worker pid=16 ppid=5 worker=0
2018-06-15 21:38:57 +0000 [info]: #0 following tail of /var/log/containers/kube-prometheus-exporter-node-fwnkt_prometheus_node-exporter-1412af047f962327fb4e3f7949fac5028ae156606e68d064240a78d37fd8af65.log
2018-06-15 21:38:57 +0000 [info]: #0 following tail of /var/log/containers/kube-node-drainer-ds-bghgj_kube-system_main-7a733ef08fe677ea9c3998026c6e3149b30ffbf031c9ddfba8450dcb9ce8dae6.log
2018-06-15 21:38:57 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
```
My config:
```
@type null
@type tail
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag kubernetes.*
format json
read_from_head true
@type kubernetes_metadata
@type record_transformer
enable_ruby true
kubehost ${record.fetch("kubernetes", Hash.new).fetch("host", "unknown_host")}
@type cloudwatch_logs
log_group_name "#{ENV['LOG_GROUP_NAME']}"
log_stream_name_key kubehost
remove_log_group_name_key true
auto_create_stream true
put_log_events_retry_limit 20
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the crash with the v1.2-debian-cloudwatch image using the posted fluent.conf and compare it with v0.12-debian-cloudwatch on Kubernetes 1.8.13 and CoreOS 1745.5.0. Inspect the container termination output beyond the startup log and compare the configured fluentd and plugin versions; done means identifying the regression or actionable failure and confirming the daemonset remains running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, ruby
- Domain
- cloud, devops, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100