opensearch-project / opensearch-project/data-prepper

[BUG] Docker container does not stop running after Data Prepper shuts down

Open
#3,141 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
I'm using the Docker version of Data Prepper (opensearch-data-prepper:2.4.0-SNAPSHOT) built from source (./gradlew :dockerSolution:buildDockerImages). When using a pipeline that includes an Opensearch source, invoking Data Prepper's shutdown API appears to terminate the Data Prepper server but does not stop the Docker container.

Snippet of Data Prepper logs:

...
Found openjdk version  of 17.0
2023-08-10T23:16:46,589 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [opensearch] as source component for the pipeline [test-pipeline]
....
2023-08-10T23:16:46,940 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [opensearch] as sink component
....
2023-08-10T23:16:47,223 [main] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Initiating pipeline execution
....
2023-08-10T23:16:47,488 [main] INFO  org.opensearch.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server running at :4900
....
2023-08-10T23:16:48,143 [test-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Submitting request to initiate the pipeline processing
....
2023-08-10T23:16:51,810 [pool-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Received shutdown signal with processor shutdown timeout PT30S and sink shutdown timeout PT30S. Initiating the shutdown process
....
2023-08-10T23:17:21,813 [pool-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Shutting down processor process workers.
....
2023-08-10T23:17:23,026 [pool-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Shutting down sink process workers.
2023-08-10T23:17:23,027 [pool-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Pipeline fully shutdown.
2023-08-10T23:17:23,057 [pool-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server stopped

Command line:

$ docker ps
CONTAINER ID   IMAGE                                    COMMAND                  CREATED          STATUS          PORTS                    NAMES
08319e699cae   opensearch-data-prepper:2.4.0-SNAPSHOT   "/entrypoint.sh bin/…"   23 minutes ago   Up 22 minutes   0.0.0.0:4900->4900/tcp   epic_jackson
$ docker top 0831
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                48665               48639               3                   23:16               ?                   00:00:43            java -Dlog4j.configurationFile=/usr/share/data-prepper/config/log4j2-rolling.properties -Ddata-prepper.dir=/usr/share/data-prepper -cp /usr/share/data-prepper/lib/* org.opensearch.dataprepper.DataPrepperExecute
root                49046               48639               0                   23:19               ?                   00:00:00            /bin/bash

Pipeline configuration:

test-pipeline:
  sink:
  - opensearch:
      bulk_size: 2
      document_id_field: getMetadata("opensearch-document_id")
      hosts:
      - [redacted]
      index: ${getMetadata("opensearch-index")}
      password: [redacted]
      username: [redacted]
  source:
    opensearch:
      disable_authentication: true
      hosts:
      - [redacted]
      indices:
        exclude:
        - index_name_regex: \.*
        include:
        - index_name_regex: [redacted]
        - index_name_regex: [redacted]

This is possibly an issue only with the opensearch source plugin - I used the following pipeline and the Docker container stopped correctly (the sink endpoint in both cases is the same).

http-test-pipeline:
    source:
        http:
    sink:
        - opensearch:
            hosts: [redacted]
            username: [redacted]
            password: [redacted]
            index: "httptest"

To Reproduce
See above

Expected behavior
The Docker container stops after the Data Prepper server has shut down

Screenshots
N/A

Environment (please complete the following information):

  • OS: MacOS 12.6.6
  • Version: Data Prepper 2.4.0-SNAPSHOT

Additional context
N/A

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the Docker image built by ./gradlew :dockerSolution:buildDockerImages, comparing the opensearch-source pipeline with the working http-source pipeline. Inspect /entrypoint.sh and the DataPrepperExecute process shown by docker top, then trace shutdown handling for the opensearch source. Done means the container exits after the Data Prepper server reports that it has stopped.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.