huggingface / huggingface/sentence-transformers

semantic search suggestion finding relevant documents

Open
#126 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.1k
Forks
2.9k
Avg merge
1d 19h
Merged PRs (30d)
61

Description

I followed your conversion here and your explanations it really help me a lot on how to implement semantic search.

https://github.com/huggingface/transformers/issues/876

I wanted your suggestions on this further, if you could help me out.

The problem i am trying to solve is to implement semantic search on a database of candidate resumes.
So i have lot of resumes of candidate who apply to job positions, i want to be able to search and find relevant candidates effectively.

the approach i am planning to implement is elastic search (BM25) and on top of it planning to use sentence transformers for re ranking.

By questions are
a) Should i train using word embedding like word2vec (i have a custom trained word2vec on a large dataset for this) or use contextual embeddings like bert, robert ?

b) For what i looks like sentence transformers are effective when applied on smaller sentences, but a resume is a full document. so i encode the entire resume to a single vector? or maybe encode it section wise or something else?

c) is BM25 + re ranking via sentence transformer the best solution for this. or should do semantic search using fiass on the full data corpus.

Would be grateful for your help on this.

Thanks

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the linked Transformers issue #876 and the proposed Elasticsearch BM25 plus sentence-transformers approach. The issue does not identify repository files or tests; a useful outcome would first define whether to encode whole resumes or sections and whether BM25 reranking or FAISS is the intended solution.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
machine-learning, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.