opensearch-project / opensearch-project/data-prepper

OpenSearch sink not rolling over the index after upgrading from 2.7 to 2.10

Open
#5,258 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Since migrating from 2.7.0 to 2.10.1 the index id stayed at the same number (000056) and is no longer rolling over. You can see below that previously indexes would be rolled around 500mb but this latest one is now at 108.5gb. Number of documents used to be around 300000 but now it's more than 73 million. Did I miss some configuration?

  | otel-v1-apm-span-000056 | green | Yes | open | 108.5gb | 54.1gb | 73591140 | 4806948 | 1 | 1
  | otel-v1-apm-span-000055 | green | Yes | open | 471mb | 235.5mb | 231704 | 24212 | 1 | 1
  | otel-v1-apm-span-000054 | green | Yes | open | 478.3mb | 239.1mb | 299849 | 20501 | 1 | 1
  | otel-v1-apm-span-000053 | green | Yes | open | 504mb | 252mb | 310190 | 26766 |  
..

Data prepper config:

raw-pipeline:
  workers: 2
  delay: "3000"
  source:
    pipeline:
      name: "otel-trace-pipeline"
  buffer:
    bounded_blocking:
      buffer_size: 10240
      batch_size: 160
  processor:
    - delete_entries:
        with_keys: ['command_args']
    - otel_traces:
    - otel_trace_group:
        hosts: ["https://opensearch-node1:9200"]
        username: admin
        password: ------
        insecure: true
  sink:
    - opensearch:
        hosts: ["https://opensearch-node1:9200"]
        index_type: trace-analytics-raw
        username: admin
        password: --------
        insecure: true

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied Data Prepper configuration with the OpenSearch sink and compare index rollover behavior between versions 2.7.0 and 2.10.1. Trace the trace-analytics-raw index rollover path and verify that indexes again roll over near the previously observed 500 MB size and document count.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.