opensearch-project / opensearch-project/data-prepper

data is not stored in the format of [<metric_name>: <value>] in OpenSearch.

Open
#4,281 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Versions:
opensearch & opensearch dashboard 2.11.1
data-prepper 2.6.1
opentelemetry-collector-contrib 0.92.0

When utilizing DataPrepper's metric pipeline to transmit data collected through OpenTelemetry to OpenSearch, the data is not stored in the format of <metric_name> : <value> in OpenSearch.

OTLE collector to collect kubelet data.
If you check in the opensearch dashboard, the metric name is stored in the name field.

{
   name: <metric_name>
   value: <value>
}

It seems to save as

I want to easily use the visualization features you provide.

{ 
    <metric_name>: <value>
}

, is it possible to change it to something like this?

[name field]
스크린샷 2024-03-14 오후 3 40 48

[data-prepper config]
스크린샷 2024-03-14 오후 3 43 34

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the metric pipeline with the listed Data Prepper, OpenTelemetry Collector, and OpenSearch versions, using the attached Data Prepper configuration. Inspect the resulting OpenSearch document and determine whether the metric pipeline or stored document mapping controls the name/value shape. Done means the requested metric-name-to-value representation is stored and can be used by OpenSearch visualizations.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.