opensearch-project / opensearch-project/data-prepper

[BUG] Custom ISM policy isnt injected

Open
#4,450 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug question
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
On fresh start (DataPrepper against fresh empty initialized OpenSearch DB) we expecting, that DataPrepper inject our custom ISM policy + custom index template.

From logs from DataPrepper I can see, that the index template is managed, but ISM policy not.

Expected behavior
If custom ISM policy is used, must be showed in ISM management GUI and ISM must managing the OTel indices.

Environment (please complete the following information):

  • DataPrepper Docker image v 2.6.2
  • OpenSearch Docker image v2.11.1

Additional context

/usr/share/data-prepper/config/data-prepper-config.yaml

ssl: false

/usr/share/data-prepper/otel-span-index-template.json

{
  "index_patterns": ["otel-v1-apm-span-*"],
  "version": 1,
  "template": {
    "settings": {
      "plugins.index_state_management.rollover_alias": "otel-v1-apm-span"
    }
  }
}

/usr/share/data-prepper/otel-span-ism-policy.json

{
  "policy": {
    "policy_id": "otel-span",
    "description": "Managing raw spans for trace analytics",
    "default_state": "current_write_index",
    "states": [
      {
        "name": "current_write_index",
        "actions": [
            {
                "rollover": {
                    "min_size": "10gb",
                    "min_index_age": "24h"
                }
            }
        ],
        "transitions": [
            {
                "state_name": "delete",
                "conditions": {
                    "min_index_age": "3d"
                }
            }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ]
      }
    ],
    "ism_template": [
      {
        "index_patterns": ["otel-v1-apm-span-*"]
      }
    ]
  }
}

/usr/share/data-prepper/pipelines/pipelines.yaml

entry-pipeline:
  workers: 4
  delay: "100"
  source:
    otel_trace_source:
      ssl: true
      sslKeyCertChainFile: "/usr/share/data-prepper/server.crt"
      sslKeyFile: "/usr/share/data-prepper/server.key"
      port: 21890
      authentication:
        http_basic:
          username: REDACTED
          password: REDACTED
  buffer:
    bounded_blocking:
      buffer_size: 1024
      batch_size: 256
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"

raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - otel_traces:
  sink:
    - opensearch:
        #index_type: trace-analytics-raw
        index_type: custom
        index: otel-v1-apm-span
        hosts: [ https://os-endpoint.REDACTED:9200 ]
        cert: "/usr/share/data-prepper/root-ca.pem"
        template_file: "/usr/share/data-prepper/otel-span-index-template.json"
        ism_policy_file: "/usr/share/data-prepper/otel-span-ism-policy.json"
        username: REDACTED
        password: REDACTED

service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - service_map:
  sink:
    - opensearch:
        index_type: trace-analytics-service-map
        hosts: [ https://os-endpoint.REDACTED:9200 ]
        cert: "/usr/share/data-prepper/root-ca.pem"
        username: REDACTED
        password: REDACTED

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the fresh-start setup with DataPrepper 2.6.2 and OpenSearch 2.11.1 using pipelines.yaml, the custom otel-span-index-template.json, and otel-span-ism-policy.json. Check the OpenSearch sink's handling of ism_policy_file and its logs, then verify that the custom policy appears in ISM management and manages the OTel indices.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.