MaartenGr / MaartenGr/BERTopic

Is it possible to use separate word and document embedding models?

Open
#1,781 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

I'm wondering if it's possible to use separate models for word and document embeddings with BERTopic. Does something break if I pass it an embedding model that treats words and documents differently (essentially, embeds them in a different space, same or different dimensionality)? Let's say my documents are all more than one word, so the embedder could make the distinction. I see a few possible answers:

1. Yes, it's possible and makes sense.
2. It would possibly make sense, but not currently implemented (may be easy to modify BERTopic to do so, though—I'm not afraid!)
3. No, and it's fundamentally not possible due to BERTopic requiring words and documents to be embedded in the same space (why?)

Do I understand correctly that word embeddings are essentially only used for the topic representation?

My motivation:

I'm working on a corpus of documents from a specific context. They contain things like acronyms and identifiers that are not commonly known. I would love to fine-tune some BERT model to give me customized word embeddings that capture the relationships of the vocabulary in my context.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by tracing BERTopic's document-embedding and topic-representation paths, then determine whether separate word and document spaces are supported or would require a design change. Done means reaching a clear, documented answer about feasibility and the required implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.