replicatedhq / replicatedhq/kURL
Fluentd config to Parse the log field correctly
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 809
- Forks
- 81
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 20
Description
With the default fluentd config, the fields like logger, severity, tenant-id, trace-id and all other fields that are part of json log field are all crammed into one big json blob and hence not queriable.
To index these fields correctly so that all the keys which are part of log field which is a json are queriable, we need to modify the fluentd config like this and right now there isn't any mechanism through which we can pass this filter.
Can you guys please add an additional option through which this config is automatically injected at the time of Infra setup so that one additional manual step is avoided
step1:
kubectl -n logging edit configmap fluentdconf
# Fixes json fields in Elasticsearch
<filter kubernetes.**>
@type parser
format json
key_name log
reserve_time true
reserve_data true
remove_key_name_field true
emit_invalid_record_to_error false
</filter>
step2: change some param like requests or limits so that pods that are under daemonset gets restarted with updated configMap
kubectl -n logging edit ds/fluentd
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Infra setup path that creates the fluentdconf ConfigMap and fluentd DaemonSet. Compare it with the parser filter and kubectl steps in the issue, then verify a fresh setup injects the filter and rolls out the updated configuration without manual edits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, kubernetes, shell
- Domain
- devops, infrastructure, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100