deepset-ai / deepset-ai/haystack

Enable grouping of pipelines via Sub-pipelines

Open
#9,091 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

P2
Dominant language
Python
Stars
26.6k
Forks
3.2k
Avg merge
1d 3h
Merged PRs (30d)
194

Description

Is your feature request related to a problem? Please describe.
Problem statement: Pipelines can grow and you want to have certain grouping (e.g. sub tasks like retrieval) When creating the pipeline you want to reduce the mental load by solving smaller challenges (e.g. retrieval) and then move on with the next part. Additionally this part might stay the same across multiple pipelines and you want to reuse it.

Motivation for "sub-pipelines". When we shared haystack with the customers - providing only two concepts pipelines and components resonated very well with them.

Describe the solution you'd like
Why do we limit ourselves to only connecting components and allow pipelines to be used within the add_component call (rename to just “add”). This would allow grouping, reusing pipeline templates within projects and having less concepts to learn that not generally apply to how graph execution engines work.

Describe alternatives you've considered
SuperComponents, containing again pipelines.

Additional context
Pseudo code might look like this:

from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.core.pipeline import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore

# Random pipeline handling any sort of retrieval
retriever_pipeline = Pipeline()
retriever_pipeline.add("retriever", InMemoryBM25Retriever(document_store=InMemoryDocumentStore()))


# Rag pipeline that uses a generic retriever pipeline
rag_pipeline = Pipeline()
rag_pipeline.add("retriever_pipeline", retriever_pipeline)  # add_component => add
rag_pipeline.add("llm", OpenAIChatGenerator())
rag_pipeline.add("prompt_builder", ChatPromptBuilder(template=...))
rag_pipeline.connect(
    "retriever_pipeline.documents", "prompt_builder.documents"
)  # connect a pipelines output to another component input of a component
rag_pipeline.connect("prompt_builder", "llm")

Serialized version might look like this:

# output after calling rag_pipeline.dumps()
components:
  llm:
    type: haystack.components.generators.chat.OpenAIChatGenerator
    init_parameters:
      model_name_or_path: gpt-4o-mini-2024-07-18
  prompt_builder:
    type: haystack.components.builders.ChatPromptBuilder
    init_parameters:
      template: ...

pipelines:
  - name: retriever_pipeline
    components:
      - retriever:
          type: haystack.components.retrievers.in_memory.InMemoryBM25Retriever
          init_parameters:
            document_store:
              type: haystack.document_stores.in_memory.InMemoryDocumentStore
              ....

connections:
  - receiver: pipelines.retriever_pipeline.documents # Optional pipelines prefix if name ambiguous
    sender: components.prompt_builder.documents # Optional sender prefix if name ambiguous
  - receiver: llm.documents
    sender: prompt_builder.documents

max_loops_allowed: 100
metadata: {}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing Pipeline API, especially add_component/add, connect, and dumps, then compare their behavior with the nested retriever_pipeline examples in the issue. The work is done when a Pipeline can be added and connected as a sub-pipeline and the resulting structure can be serialized in the requested form, including nested components and connections.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.