opensearch-project / opensearch-project/data-prepper
Data Prepper servers do not process requests at all if pipeline is inoperable at startup
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
When Data Prepper is starting, if a sink fails to initialize, it does not start the source. This is by design to avoid taking requests if the server may not come up. However, one downside with the current implementation is that the server itself does not respond. Ideally, the server could be up, but unable to accept requests.
Describe the solution you'd like
When Data Prepper is starting, start servers but do not allow them to accept requests. Instead, they could respond with 503 Service Unavailable.
To implement, this might require some API changes to Source to allow the pipeline to start before it is ready. A new void start(); method could be used to perform any start-up work before the Buffer is available.
This can apply for all the HTTP/gRPC sources.
Describe alternatives you've considered (Optional)
Perhaps the server could return a 4xx response, but none seem quite right.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Source startup lifecycle, including how the Buffer becomes available, across the HTTP and gRPC sources. The change is done when servers start during pipeline initialization but reject requests with 503 Service Unavailable until the pipeline is ready.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100