opensearch-project / opensearch-project/data-prepper

[BUG] Data Prepper starts without a source

Open
#1,829 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
I started data prepper with a generic log pipeline that used AWS ACM certificate provider for the HTTP log source. However, I gave an incorrect certificate ARN that data prepper was unable to use. Data prepper still came up successfully however I was unable to send log data because the http source was not running.

2022-09-28T10:27:51.174-05:00 | CopyException retrieving the certificate with arn: arn:aws:acm:us-east-1:123456789012:certificate/someCertificate 
2022-09-28T10:27:51.174-05:00 | software.amazon.awssdk.services.acm.model.ResourceNotFoundException: Could not find certificate arn:aws:acm:us-east-1:123456789012:certificate/someCertificate . (Service: Acm, Status Code: 400, Request ID: 73995789-64ce-4e90-9c3c-20649fd6cdac)
2022-09-28T10:27:51.174-05:00 | at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125) ~[data-prepper-core-1.5.0.jar:1.5.0]
...
2022-09-28T10:27:51.180-05:00 | at com.amazon.dataprepper.plugins.certificate.acm.ACMCertificateProvider.getCertificate(ACMCertificateProvider.java:90) ~[data-prepper-core-1.5.0.jar:1.5.0]
2022-09-28T10:27:51.180-05:00 | at com.amazon.dataprepper.plugins.source.loghttp.HTTPSource.start(HTTPSource.java:83) ~[data-prepper-core-1.5.0.jar:1.5.0]
...
2022-09-28T10:27:51.189-05:00 | Data Prepper server running at :4900

To Reproduce
PipelineConfiguration.yaml snippet:

log-pipeline:
 source:
    http:
      acm_certificate_arn: "arn: arn:aws:acm:us-east-1:123456789012:certificate/someCertificate "
      aws_region: "us-east-1"
      port: 21890
      use_acm_certificate_for_ssl: true
      ssl: true
      authentication:
        ...
  processor: ...

Expected behavior
I expected data prepper to fail to start if there is no source. Running data prepper without a source is useless. This may not be different if data prepper would support multiple sources.

Additional context
This is similar to: #936

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ACMCertificateProvider.java and HTTPSource.java, the entry points named in the stack trace, and trace how the certificate failure is handled during startup. Check the server startup path next. Done means Data Prepper does not report a running server when the HTTP source cannot start because its certificate is unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.