MaartenGr / MaartenGr/BERTopic
`transform` method not handling single embeddings or strings given to it.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
**Issue**
Currently the `transform` method of the `BERTopic` has to recieve a list of strings as the documents and if embeddings are present then it needs to be a 2darray of shape (len(documents, embedding_dimension). This requires extra reshape calls when trying to call the `transform` method on a single document.
For example it would be nicer to do this:
```python
document = 'This is a really interesting document'
document_embedding = np.random.rand(768)
topic_model.transform(document, document_embedding)
```
Example of what you currently have to do
```python
document = 'This is a really interesting document'
document_embedding = np.random.rand(768)
topic_model.transform([document], document_embedding.reshape(1,-1))
```
Currently if you run it you get this error.
```
BFile ~/code/BERTopic/bertopic/_utils.py:55, in check_embeddings_shape(embeddings, docs)
else:
if embeddings.shape[0] != len(docs):
---> raise ValueError("Make sure that the embeddings are a numpy array with shape: "
"(len(docs), vector_dim) where vector_dim is the dimensionality "
"of the vector embeddings. ")
ValueError: Make sure that the embeddings are a numpy array with shape: (len(docs), vector_dim) where vector_dim is the dimensionality of the vector embeddings.
```
**Why I think it is happening**
This is because the embeddings shape check inside `transform` happens before:
1. the `document` argument is converted into a list if it is str
2. The embedding is reshaped from a 1d array to 2d array _(Note that it currently never does this)._
**Thoughts on making it better**
Am I missing something here about np arrays or the transform function? It seems to me that wanting to transform a single document is common enough that the function could have that little bit of extra functionality.
I have already made the change on a fork for my uses did you agree with the idea and would you like a PR?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with BERTopic.transform and the embeddings shape check identified at bertopic/_utils.py:55, tracing the order in which documents and embeddings are normalized. Done means a single string with a one-dimensional embedding is accepted while the existing list-of-strings and two-dimensional embedding behavior remains valid; add or update tests where the transform behavior is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100