deepset-ai / deepset-ai/haystack
Enable grouping of pipelines via Sub-pipelines
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
Is your feature request related to a problem? Please describe.
Problem statement: Pipelines can grow and you want to have certain grouping (e.g. sub tasks like retrieval) When creating the pipeline you want to reduce the mental load by solving smaller challenges (e.g. retrieval) and then move on with the next part. Additionally this part might stay the same across multiple pipelines and you want to reuse it.
Motivation for "sub-pipelines". When we shared haystack with the customers - providing only two concepts pipelines and components resonated very well with them.
Describe the solution you'd like
Why do we limit ourselves to only connecting components and allow pipelines to be used within the add_component call (rename to just “add”). This would allow grouping, reusing pipeline templates within projects and having less concepts to learn that not generally apply to how graph execution engines work.
Describe alternatives you've considered
SuperComponents, containing again pipelines.
Additional context
Pseudo code might look like this:
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.core.pipeline import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
# Random pipeline handling any sort of retrieval
retriever_pipeline = Pipeline()
retriever_pipeline.add("retriever", InMemoryBM25Retriever(document_store=InMemoryDocumentStore()))
# Rag pipeline that uses a generic retriever pipeline
rag_pipeline = Pipeline()
rag_pipeline.add("retriever_pipeline", retriever_pipeline) # add_component => add
rag_pipeline.add("llm", OpenAIChatGenerator())
rag_pipeline.add("prompt_builder", ChatPromptBuilder(template=...))
rag_pipeline.connect(
"retriever_pipeline.documents", "prompt_builder.documents"
) # connect a pipelines output to another component input of a component
rag_pipeline.connect("prompt_builder", "llm")
Serialized version might look like this:
# output after calling rag_pipeline.dumps()
components:
llm:
type: haystack.components.generators.chat.OpenAIChatGenerator
init_parameters:
model_name_or_path: gpt-4o-mini-2024-07-18
prompt_builder:
type: haystack.components.builders.ChatPromptBuilder
init_parameters:
template: ...
pipelines:
- name: retriever_pipeline
components:
- retriever:
type: haystack.components.retrievers.in_memory.InMemoryBM25Retriever
init_parameters:
document_store:
type: haystack.document_stores.in_memory.InMemoryDocumentStore
....
connections:
- receiver: pipelines.retriever_pipeline.documents # Optional pipelines prefix if name ambiguous
sender: components.prompt_builder.documents # Optional sender prefix if name ambiguous
- receiver: llm.documents
sender: prompt_builder.documents
max_loops_allowed: 100
metadata: {}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing Pipeline API, especially add_component/add, connect, and dumps, then compare their behavior with the nested retriever_pipeline examples in the issue. The work is done when a Pipeline can be added and connected as a sub-pipeline and the resulting structure can be serialized in the requested form, including nested components and connections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100