deepset-ai / deepset-ai/haystack
Provide relevant schema of pipeline serialization
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
Is your feature request related to a problem? Please describe.
It's great that this library supports serializing the pipelines into a human editable format as that's a very useful feature that other packages do not have. However, I see an issue with the provided format
There is no defined schema for what fields each component needs to have set. This is important because otherwise a user purely editing the YAML will not know what fields are supported and we can't introduce any validation. I am also planning to use JSON format instead with a custom marshaller and provide a JSON schema which can be edited as a GUI since there are plenty of JSON Schema GUI editors other there. However, this requires me to manually inspect each component and come up with a JSON schema which would be a nightmare in terms of maintenance when components change
Describe the solution you'd like
I would like to propose that the haystack framework provide a way to generate a JSON schema for all the components that basically captures that input format which will allow us to validate and build on top. Is this possible in the current design?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the pipeline serialization entry points and the component definitions that determine YAML fields. Read how pipelines are currently serialized and how components expose their inputs, then assess how JSON Schema generation and validation would fit the existing design. Done means a maintainable schema can be generated for all components and supports validation of edited YAML or JSON.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100