opensearch-project / opensearch-project/data-prepper

No clear indication of single node vs. multi node data prepper cluster

Open
#2,413 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
Data Prepper can either be run as a single instance, or as two or more instances that essentially make up a cluster of data prepper instances.

When configuring the data-prepper-config.yaml there are certain settings that only apply in a multi-node scenario. For example, the peer forwarder (https://github.com/opensearch-project/data-prepper/blob/main/docs/peer_forwarder.md) is only relevant in a multi-node scenario. The peer forwarder is configured with service_discovery settings to allow peers to communicate and forward Events. However, the configuration for service discovery is too coupled to peer forwarding, since service_discovery can be applied to other situations when data prepper nodes need to be aware of other data prepper nodes.

Describe the solution you'd like
I am proposing we break the service discovery configuration out from under the peer_forwarder (of course, we can do this while not making breaking changes and just deprecating the old peer forwarder service discovery config. It would be going from this

peer_forwarder:
    discovery_mode: static
    static_endpoints: ["dataprepper1", "dataprepper2"]
    ... other peer forwarder configurations ... 

to configuring service_discovery at the highest level, like this

some_other_feature_that_uses_service_discovery:
peer_forwarder:
   ... other peer forwarder configurations ...
service_discovery:
    discovery_mode: static
    static_endpoints: ["dataprepper1", "dataprepper2"]

Additionally, a configuration for disovery_mode could be added to service_discovery, which will be useful for all plugins or features of data prepper that might behave differently if they are in a multi-node vs single-node setting (for example, source coordination with a single node does not require a distributed store #2412).

This configuration would be a discovery_mode of type single_instance (or single_node), which is a way to explicitly say that this will always be just a single instance of data prepper not as a part of a cluster. There are cases to make for this to be the default discovery_mode if no service_discovery is found in the data-prepper-config.yaml. If single_node is not the default, then it is always assumed to be a multi-node scenario, which will not properly work for stateful aggregations without a peer forwarder and service discovery configuration. Defaulting to single_node seems like the safer option, since that is technically how data prepper works today (no peer forwarder configuration will essentially act as single node). The only other alternative is to make it required, but this is a breaking change. The final configuration would look like this, and would illustrate to anyone that could use the information that this is only a single instance of data prepper working on its own.

service_discovery:
     discovery_mode: single_node

Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with data-prepper-config.yaml and docs/peer_forwarder.md to understand the existing peer-forwarder service discovery configuration and its documented behavior. Review the related source and configuration entry points, including the context of issue #2412; done means service discovery is represented independently and single-node versus multi-node behavior is explicit without breaking the existing configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.