MaartenGr / MaartenGr/BERTopic
Is it possible to use separate word and document embedding models?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
I'm wondering if it's possible to use separate models for word and document embeddings with BERTopic. Does something break if I pass it an embedding model that treats words and documents differently (essentially, embeds them in a different space, same or different dimensionality)? Let's say my documents are all more than one word, so the embedder could make the distinction. I see a few possible answers:
1. Yes, it's possible and makes sense.
2. It would possibly make sense, but not currently implemented (may be easy to modify BERTopic to do so, though—I'm not afraid!)
3. No, and it's fundamentally not possible due to BERTopic requiring words and documents to be embedded in the same space (why?)
Do I understand correctly that word embeddings are essentially only used for the topic representation?
My motivation:
I'm working on a corpus of documents from a specific context. They contain things like acronyms and identifiers that are not commonly known. I would love to fine-tune some BERT model to give me customized word embeddings that capture the relationships of the vocabulary in my context.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by tracing BERTopic's document-embedding and topic-representation paths, then determine whether separate word and document spaces are supported or would require a design change. Done means reaching a clear, documented answer about feasibility and the required implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100