opensearch-project / opensearch-project/data-prepper
[BUG] Data Prepper starts without a source
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
I started data prepper with a generic log pipeline that used AWS ACM certificate provider for the HTTP log source. However, I gave an incorrect certificate ARN that data prepper was unable to use. Data prepper still came up successfully however I was unable to send log data because the http source was not running.
2022-09-28T10:27:51.174-05:00 | CopyException retrieving the certificate with arn: arn:aws:acm:us-east-1:123456789012:certificate/someCertificate
2022-09-28T10:27:51.174-05:00 | software.amazon.awssdk.services.acm.model.ResourceNotFoundException: Could not find certificate arn:aws:acm:us-east-1:123456789012:certificate/someCertificate . (Service: Acm, Status Code: 400, Request ID: 73995789-64ce-4e90-9c3c-20649fd6cdac)
2022-09-28T10:27:51.174-05:00 | at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125) ~[data-prepper-core-1.5.0.jar:1.5.0]
...
2022-09-28T10:27:51.180-05:00 | at com.amazon.dataprepper.plugins.certificate.acm.ACMCertificateProvider.getCertificate(ACMCertificateProvider.java:90) ~[data-prepper-core-1.5.0.jar:1.5.0]
2022-09-28T10:27:51.180-05:00 | at com.amazon.dataprepper.plugins.source.loghttp.HTTPSource.start(HTTPSource.java:83) ~[data-prepper-core-1.5.0.jar:1.5.0]
...
2022-09-28T10:27:51.189-05:00 | Data Prepper server running at :4900
To Reproduce
PipelineConfiguration.yaml snippet:
log-pipeline:
source:
http:
acm_certificate_arn: "arn: arn:aws:acm:us-east-1:123456789012:certificate/someCertificate "
aws_region: "us-east-1"
port: 21890
use_acm_certificate_for_ssl: true
ssl: true
authentication:
...
processor: ...
Expected behavior
I expected data prepper to fail to start if there is no source. Running data prepper without a source is useless. This may not be different if data prepper would support multiple sources.
Additional context
This is similar to: #936
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ACMCertificateProvider.java and HTTPSource.java, the entry points named in the stack trace, and trace how the certificate failure is handled during startup. Check the server startup path next. Done means Data Prepper does not report a running server when the HTTP source cannot start because its certificate is unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100