opensearch-project / opensearch-project/data-prepper
Running sinks serially (with Pipeline Connector)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Hi,
I am configuring a DataPrepper pipeline in which I need two sinks, but I need to make sure that one is completed before the other one. I have two questions regarding this-
First, is it possible to configure the sinks in such a way that they run serially rather than in parallel? I've done my best to read through the documentation and it appears that the answer to this is no, but I'd like to have confirmation before completely dismissing it as an option.
Second, if the answer to the first question is no, then is it possible to set up a Pipeline Connector such that the output of the first pipeline is used as the source for the second pipeline, rather than having both pipelines use the same source? What I'm referring to here is something like this example:
input-pipeline:
source:
file:
path: path/to/input-file
processor:
- string_converter:
upper_case: true
sink:
- pipeline:
name: "output-pipeline-1"
output-pipeline-1:
source:
pipeline:
name: "input-pipeline" //use the already processed data rather than the immediate source of input-pipeline
sink:
- file:
path: path/to/output-1-file
According to the overview documentation,
Pipeline Connectors help to process data from single source in multiple pipelines
However, I'm not sure whether "single source" here means that putting input-pipeline as a source means it will use the same source already specified, or whether it will use the output of that pipeline for the source (meaning, the already processed data).
Please do let me know whether specifying the order of the sinks is possible, or, if not, help clarify the input/output workflow of the pipeline connectors. Thanks in advance!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/overview.md and the Pipeline Connector configuration documentation. Verify the documented sink execution order and whether a connector consumes processed output or shares the original source. Done means the documentation clearly answers both questions with an accurate configuration example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100