CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator
Store word models in elasticsearch?
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- Avg merge
- 3d 3m
- Merged PRs (30d)
- 9
Description
All word models logic currently happens entirely in python, with vector-related logic handled by gensim.
We might consider storing this data in elasticsearch instead. Basically, you could make a `parliament-uk-models` index to accompany `parliament-uk`, with the following fields + field types
- `date_start`: date
- `date_end`: date
- `term`: keyword
- `vector`: [dense vector](https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html)
You can use the vector field to request, say, the N nearest neighbours based on cosine similarity.
This could work a lot faster than our current approach. Elasticsearch allows you to use an HNSW algorithm which takes some time to index, but saves time during search.
@BeritJanssen , what do you think?
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the existing word-model logic in Python and the vector-related logic handled by gensim; the payload does not name specific files or tests. Compare the proposed parliament-uk-models index, its date, term, and dense-vector fields, and HNSW nearest-neighbour search with the current approach, then establish whether the Elasticsearch design is faster and complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- data, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100