[BUG] Data Prepper sinks repeatedly fail with "Request execution cancelled" on initial Docker Compose startup

What is the bug?

On a fresh local deployment using the repository's default Docker Compose configuration, OpenSearch becomes healthy and the Data Prepper container reports Up, but some Data Prepper OpenSearch sinks repeatedly fail during initialization with:

Failed to initialize OpenSearch sink with a retryable exception.
java.lang.RuntimeException: Request execution cancelled
Caused by: java.util.concurrent.CancellationException: Request execution cancelled

In my reproduction:

• traces-raw-pipeline successfully initialized.
• otel-logs-pipeline and service-map-pipeline repeatedly failed.
• The failed sinks retried approximately every 200 ms.
• Waiting did not make the affected sinks recover.
• Restarting only Data Prepper, without changing any configuration, caused all three sinks to initialize successfully.

This leaves the stack partially ready even though docker compose ps shows Data Prepper as running.

How can one reproduce the bug?

Start from a fresh OpenSearch volume.

Warning: docker compose down -v removes local test data created by this Compose project.

docker compose down -v --remove-orphans

docker compose up -d opensearch
docker compose ps opensearch

docker compose up -d data-prepper
docker compose ps opensearch data-prepper

Inspect the Data Prepper logs:

docker compose logs --no-log-prefix data-prepper |
  grep -E 'Initializing OpenSearch sink|Initialized OpenSearch sink|Failed to initialize OpenSearch sink|Request execution cancelled'

The affected pipelines repeatedly produce output similar to:

[service-map-pipeline-sink-worker-...] INFO  OpenSearchSink - Initializing OpenSearch sink
[otel-logs-pipeline-sink-worker-...] INFO  OpenSearchSink - Initializing OpenSearch sink

[service-map-pipeline-sink-worker-...] WARN  OpenSearchSink - Failed to initialize OpenSearch sink with a retryable exception.
java.lang.RuntimeException: Request execution cancelled
    at org.opensearch.client.RestClient.extractAndWrapCause(...)
    at org.opensearch.client.RestClient.performRequest(...)
    at org.opensearch.client.opensearch.cluster.OpenSearchClusterClient.getSettings(...)
    at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkISMEnabled(...)
    at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkAndCreateIndexTemplate(...)
    at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.setupIndex(...)
Caused by: java.util.concurrent.CancellationException: Request execution cancelled

The Data Prepper API still lists all configured pipelines:

docker compose exec data-prepper \
  curl -s http://localhost:4900/list

Example response:

{
  "pipelines": [
    {"name": "otlp-pipeline"},
    {"name": "otel-traces-pipeline"},
    {"name": "service-map-pipeline"},
    {"name": "traces-raw-pipeline"},
    {"name": "otel-logs-pipeline"},
    {"name": "service-metrics-cortex-pipeline"}
  ]
}

Restart only Data Prepper:

docker compose restart data-prepper

After the restart, without any configuration changes, all OpenSearch sinks initialize and the OTLP source starts successfully.

What is the expected behavior?

Once OpenSearch reports healthy and Data Prepper starts:

• All configured OpenSearch sinks should initialize successfully.
• The stack should not require a manual Data Prepper restart.
• The service status should distinguish a running container from a fully initialized Data Prepper instance.
• Telemetry should not be accepted before the required sinks are ready.

What is your host/environment?

• Host OS: macOS 26.5.1, arm64
• Docker CLI: 29.4.0
• Docker Compose: v5.1.2
• Repository commit: 6e5b56e
• OpenSearch image: opensearchproject/opensearch:3.8.0
• Data Prepper image: sgguruda62324/opensearch-data-prepper:2.16.0-SNAPSHOT-rc1
• Data Prepper platform: linux/amd64
• Default memory limits:
  • OpenSearch: 2 GiB
  • Data Prepper: 1 GiB

Are there any screenshots?

N/A. The relevant log output and reproduction commands are included above.

Do you have any additional context?

During the first startup, all three OpenSearch sinks appeared to initialize concurrently. Two HTTP 409 responses involving an already-existing .opendistro-ism-config index were also observed during that initialization period.

This may indicate a race during concurrent ISM or index-template initialization, but the root cause has not yet been confirmed. Other possibilities include the Data Prepper snapshot image itself or compatibility with OpenSearch 3.8.0.

The following behavior has been confirmed:

1. The issue is reproducible with a fresh OpenSearch volume.
2. OpenSearch is healthy when it occurs.
3. Data Prepper remains Up while some sinks continuously fail.
4. Restarting only Data Prepper resolves the problem without configuration or resource changes.

Data Prepper currently has no Docker Compose health check that verifies whether all sinks have initialized. As a result, dependent services or users may treat the stack as ready while log ingestion and service-map generation are unavailable.

Start with the repository's default Docker Compose configuration and reproduce the failure from a fresh OpenSearch volume using the commands in the issue. Read the Data Prepper sink initialization logs, including the concurrent ISM/index-template requests and HTTP 409 responses. Done means all configured sinks initialize without a manual restart and readiness does not report success before they are available.