huggingface / huggingface/sentence-transformers
semantic search suggestion finding relevant documents
- Dominant language
- Python
- Stars
- 19.1k
- Forks
- 2.9k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 61
Description
I followed your conversion here and your explanations it really help me a lot on how to implement semantic search.
https://github.com/huggingface/transformers/issues/876
I wanted your suggestions on this further, if you could help me out.
The problem i am trying to solve is to implement semantic search on a database of candidate resumes.
So i have lot of resumes of candidate who apply to job positions, i want to be able to search and find relevant candidates effectively.
the approach i am planning to implement is elastic search (BM25) and on top of it planning to use sentence transformers for re ranking.
By questions are
a) Should i train using word embedding like word2vec (i have a custom trained word2vec on a large dataset for this) or use contextual embeddings like bert, robert ?
b) For what i looks like sentence transformers are effective when applied on smaller sentences, but a resume is a full document. so i encode the entire resume to a single vector? or maybe encode it section wise or something else?
c) is BM25 + re ranking via sentence transformer the best solution for this. or should do semantic search using fiass on the full data corpus.
Would be grateful for your help on this.
Thanks
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the linked Transformers issue #876 and the proposed Elasticsearch BM25 plus sentence-transformers approach. The issue does not identify repository files or tests; a useful outcome would first define whether to encode whole resumes or sections and whether BM25 reranking or FAISS is the intended solution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- machine-learning, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100