opensearch-project / opensearch-project/data-prepper
[BUG] Custom ISM policy isnt injected
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
On fresh start (DataPrepper against fresh empty initialized OpenSearch DB) we expecting, that DataPrepper inject our custom ISM policy + custom index template.
From logs from DataPrepper I can see, that the index template is managed, but ISM policy not.
Expected behavior
If custom ISM policy is used, must be showed in ISM management GUI and ISM must managing the OTel indices.
Environment (please complete the following information):
- DataPrepper Docker image v 2.6.2
- OpenSearch Docker image v2.11.1
Additional context
/usr/share/data-prepper/config/data-prepper-config.yaml
ssl: false
/usr/share/data-prepper/otel-span-index-template.json
{
"index_patterns": ["otel-v1-apm-span-*"],
"version": 1,
"template": {
"settings": {
"plugins.index_state_management.rollover_alias": "otel-v1-apm-span"
}
}
}
/usr/share/data-prepper/otel-span-ism-policy.json
{
"policy": {
"policy_id": "otel-span",
"description": "Managing raw spans for trace analytics",
"default_state": "current_write_index",
"states": [
{
"name": "current_write_index",
"actions": [
{
"rollover": {
"min_size": "10gb",
"min_index_age": "24h"
}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "3d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
]
}
],
"ism_template": [
{
"index_patterns": ["otel-v1-apm-span-*"]
}
]
}
}
/usr/share/data-prepper/pipelines/pipelines.yaml
entry-pipeline:
workers: 4
delay: "100"
source:
otel_trace_source:
ssl: true
sslKeyCertChainFile: "/usr/share/data-prepper/server.crt"
sslKeyFile: "/usr/share/data-prepper/server.key"
port: 21890
authentication:
http_basic:
username: REDACTED
password: REDACTED
buffer:
bounded_blocking:
buffer_size: 1024
batch_size: 256
sink:
- pipeline:
name: "raw-pipeline"
- pipeline:
name: "service-map-pipeline"
raw-pipeline:
source:
pipeline:
name: "entry-pipeline"
processor:
- otel_traces:
sink:
- opensearch:
#index_type: trace-analytics-raw
index_type: custom
index: otel-v1-apm-span
hosts: [ https://os-endpoint.REDACTED:9200 ]
cert: "/usr/share/data-prepper/root-ca.pem"
template_file: "/usr/share/data-prepper/otel-span-index-template.json"
ism_policy_file: "/usr/share/data-prepper/otel-span-ism-policy.json"
username: REDACTED
password: REDACTED
service-map-pipeline:
delay: "100"
source:
pipeline:
name: "entry-pipeline"
processor:
- service_map:
sink:
- opensearch:
index_type: trace-analytics-service-map
hosts: [ https://os-endpoint.REDACTED:9200 ]
cert: "/usr/share/data-prepper/root-ca.pem"
username: REDACTED
password: REDACTED
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the fresh-start setup with DataPrepper 2.6.2 and OpenSearch 2.11.1 using pipelines.yaml, the custom otel-span-index-template.json, and otel-span-ism-policy.json. Check the OpenSearch sink's handling of ism_policy_file and its logs, then verify that the custom policy appears in ISM management and manages the OTel indices.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100