replicatedhq / replicatedhq/kURL

Fluentd config to Parse the log field correctly

Open
#458 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type::feature
Dominant language
Shell
Stars
809
Forks
81
Avg merge
1d 21h
Merged PRs (30d)
20

Description

With the default fluentd config, the fields like logger, severity, tenant-id, trace-id and all other fields that are part of json log field are all crammed into one big json blob and hence not queriable.

To index these fields correctly so that all the keys which are part of log field which is a json are queriable, we need to modify the fluentd config like this and right now there isn't any mechanism through which we can pass this filter.

Can you guys please add an additional option through which this config is automatically injected at the time of Infra setup so that one additional manual step is avoided

step1:
kubectl -n logging edit configmap fluentdconf
    # Fixes json fields in Elasticsearch
    <filter kubernetes.**>
      @type parser
      format json
      key_name log
      reserve_time true
      reserve_data true
      remove_key_name_field true
      emit_invalid_record_to_error false
    </filter>
step2: change some param like requests or limits so that pods that are under daemonset gets restarted with updated configMap
kubectl -n logging edit ds/fluentd

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Infra setup path that creates the fluentdconf ConfigMap and fluentd DaemonSet. Compare it with the parser filter and kubectl steps in the issue, then verify a fresh setup injects the filter and rolls out the updated configuration without manual edits.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, kubernetes, shell
Domain
devops, infrastructure, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.