opensearch-project / opensearch-project/data-prepper
OpenSearch sink not rolling over the index after upgrading from 2.7 to 2.10
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Since migrating from 2.7.0 to 2.10.1 the index id stayed at the same number (000056) and is no longer rolling over. You can see below that previously indexes would be rolled around 500mb but this latest one is now at 108.5gb. Number of documents used to be around 300000 but now it's more than 73 million. Did I miss some configuration?
| otel-v1-apm-span-000056 | green | Yes | open | 108.5gb | 54.1gb | 73591140 | 4806948 | 1 | 1
| otel-v1-apm-span-000055 | green | Yes | open | 471mb | 235.5mb | 231704 | 24212 | 1 | 1
| otel-v1-apm-span-000054 | green | Yes | open | 478.3mb | 239.1mb | 299849 | 20501 | 1 | 1
| otel-v1-apm-span-000053 | green | Yes | open | 504mb | 252mb | 310190 | 26766 |
..
Data prepper config:
raw-pipeline:
workers: 2
delay: "3000"
source:
pipeline:
name: "otel-trace-pipeline"
buffer:
bounded_blocking:
buffer_size: 10240
batch_size: 160
processor:
- delete_entries:
with_keys: ['command_args']
- otel_traces:
- otel_trace_group:
hosts: ["https://opensearch-node1:9200"]
username: admin
password: ------
insecure: true
sink:
- opensearch:
hosts: ["https://opensearch-node1:9200"]
index_type: trace-analytics-raw
username: admin
password: --------
insecure: true
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the supplied Data Prepper configuration with the OpenSearch sink and compare index rollover behavior between versions 2.7.0 and 2.10.1. Trace the trace-analytics-raw index rollover path and verify that indexes again roll over near the previously observed 500 MB size and document count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100