MaartenGr / MaartenGr/BERTopic

Huggingface transformer does not load as expected

Open
#1,952 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hello,

Instead of using `sentence-transformers/all-MiniLM-L6-v2`, I wanted to try out a custom embedding model from Huggingface. I read previous opened and closed issues and found the following approaches:

![Screenshot from 2024-04-27 13-20-17](https://github.com/MaartenGr/BERTopic/assets/32801760/c3b27a76-787a-4147-8935-ed11b0e2d348)

Code:

```python
# romanian_embedding_model = pipeline("feature-extraction", model="readerbench/RoBERT-large", tokenizer="readerbench/RoBERT-large")
# tokenizer_kwargs = {'padding':True,'truncation':True,'max_length':512,'return_tensors':'pt'}
# self.bertopic_model = BERTopic(embedding_model=lambda x: romanian_embedding_model(x, **tokenizer_kwargs), verbose=True, nr_topics='auto', n_gram_range=(1, 2))
# topics_1, probs_1 = self.bertopic_model.fit_transform(self.nontruncated_documents)
embedding_model = AutoModel.from_pretrained("dumitrescustefan/bert-base-romanian-cased-v1")
tokenizer = AutoTokenizer.from_pretrained("dumitrescustefan/bert-base-romanian-cased-v1")
romanian_embedding_model = pipeline("feature-extraction", model=embedding_model, tokenizer=tokenizer)
tokenizer_kwargs = {'padding':True, 'truncation':True, 'max_length':512, 'return_tensors':'pt'}
self.bertopic_model = BERTopic(embedding_model=lambda x: romanian_embedding_model(x, **tokenizer_kwargs), verbose=True, nr_topics='auto', n_gram_range=(1, 2))
topics_1, probs_1 = self.bertopic_model.fit_transform(self.nontruncated_documents)
```

However, by using a logger like

```python
import logging
logging.basicConfig()
logger = logging.getLogger('Something')
logger.setLevel(logging.INFO)
```

... I discovered that BERTopic doesn't actually load the Huggingface model. It simply loads `sentence-transformers/all-MiniLM-L6-v2`. Why is this the case?

Proof:

![Screenshot from 2024-04-27 13-19-24](https://github.com/MaartenGr/BERTopic/assets/32801760/6e595cce-13c0-431f-8138-e5e269af8e9b)

I will use the multilingual version until this issue is fixed.

Thank you for your patience 💯

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the shown BERTopic setup with the Hugging Face pipeline, AutoModel, and AutoTokenizer while observing the logger output. Trace how BERTopic handles the callable passed as embedding_model and determine whether the custom Romanian model is actually selected; done means the supplied model is used instead of sentence-transformers/all-MiniLM-L6-v2.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.