deepset-ai / deepset-ai/haystack
Add New `SemanticResolver` component
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
Is your feature request related to a problem? Please describe.
When generating your own knowledge graph using LLMs, the LLMMetadataExtractor component can be used to encode entities and relationships inside the metadata of haystack Document objects. LLMMetadataExtractor supports ChatGenerator's that could use structured output to define possible graph ontology, however it is common when creating knowledge to not have predefined possible entities, especially when working with large amounts of data.
This means you can have generated metadata for document_one and document_two such that;
document_one.meta = {'entities': [{'entity': 'deepset', 'entity_type': 'company'}]}
document_two.meta = {'entities': [{'entity': 'deepset GmbH', 'entity_type': 'company'}]}
when these entities are referring to the same concept. These entities are often ingested into a graph as separate nodes.
Describe the solution you'd like
Implement a new SemanticResolver component that uses the text-embedding-inference (TEI) similarity endpoint along with a given threshold to determine whether entities are similar enough to merge and then merges the entities.
Describe alternatives you've considered
Using spaCy to determine similarity, however this is much slower than TEI. However, another component could be created that utilises spaCy.
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating existing component patterns for metadata extraction and the Document metadata structure, then review the TEI similarity endpoint linked in the issue. Define how the SemanticResolver applies its threshold and merges matching entities, and verify the behavior with focused component tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100