MaartenGr / MaartenGr/BERTopic
ValueError: Distance matrix cannot contain negative values
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
First off, massive congrats and a huge thank you for the library - I can't even express in words how good it is ;)
I'm working on a fairly large (~388k) multilingual dataset and am encountering a strange error when calculating the hierarchical topics relate to the distance matrix.
Here is my code:
`texts = data['text'].astype(str).to_list()
ids = data.index.to_list()
sentence_model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2", cache_folder=cache_folder)
embeddings = sentence_model.encode(texts, show_progress_bar=True)
np.save(f'{results_path}/embeddings', embeddings)
umap_model = UMAP(
n_neighbors=100,
n_components=5,
min_dist=0.01,
metric="cosine",
low_memory=True,
n_jobs=5,
random_state=42,
)
sws = list(get_stopwords(language='multi'))
vectorizer_model = CountVectorizer(stop_words=sws, ngram_range=(1,2), min_df=10)
bertopic_args = {
"language":'multilingual',
"embedding_model": sentence_model,
"vectorizer_model": vectorizer_model,
"umap_model": umap_model,
"verbose":True,
"low_memory":True,
"calculate_probabilities":False,
}
bertopic_model = BERTopic(**bertopic_args)
topic, probs = bertopic_model.fit_transform(texts, embeddings)
test_tuned = bertopic_model.get_topic_info()
data['topic'] = topic
hierarchical_topics = bertopic_model.hierarchical_topics(texts)
`
Here the error I get:
ValueError Traceback (most recent call last)
File <timed exec>:2
File....../lib/python3.11/site-packages/bertopic/_bertopic.py:939, in BERTopic.hierarchical_topics(self, docs, linkage_function, distance_function)
937 embeddings = self.c_tf_idf_[self._outliers:]
938 X = distance_function(embeddings)
--> 939 X = validate_distance_matrix(X, embeddings.shape[0])
941 # Use the 1-D condensed distance matrix as an input instead of the raw distance matrix
942 Z = linkage_function(X)
File ..../lib/python3.11/site-packages/bertopic/_utils.py:141, in validate_distance_matrix(X, n_samples)
139 # Make sure its entries are non-negative
140 if np.any(X < 0):
--> 141 raise ValueError("Distance matrix cannot contain negative values.")
143 return X
ValueError: Distance matrix cannot contain negative values.
This code works fine for all the other datasets I've used it on.
Am I doing something wrong, or is there some issue?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the report with the supplied BERTopic, UMAP, CountVectorizer, and embedding setup. Trace BERTopic.hierarchical_topics in bertopic/_bertopic.py into validate_distance_matrix in bertopic/_utils.py and inspect the distance matrix produced for this dataset. Done means the cause of the negative values is identified and hierarchical_topics handles the reported case without raising this error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python, scikit-learn
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100