opensearch-project / opensearch-project/data-prepper

Data Prepper servers do not process requests at all if pipeline is inoperable at startup

Open
#3,309 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.

When Data Prepper is starting, if a sink fails to initialize, it does not start the source. This is by design to avoid taking requests if the server may not come up. However, one downside with the current implementation is that the server itself does not respond. Ideally, the server could be up, but unable to accept requests.

Describe the solution you'd like

When Data Prepper is starting, start servers but do not allow them to accept requests. Instead, they could respond with 503 Service Unavailable.

To implement, this might require some API changes to Source to allow the pipeline to start before it is ready. A new void start(); method could be used to perform any start-up work before the Buffer is available.

This can apply for all the HTTP/gRPC sources.

Describe alternatives you've considered (Optional)

Perhaps the server could return a 4xx response, but none seem quite right.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Source startup lifecycle, including how the Buffer becomes available, across the HTTP and gRPC sources. The change is done when servers start during pipeline initialization but reject requests with 503 Service Unavailable until the pipeline is ready.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.