MaartenGr / MaartenGr/BERTopic
"doc_length" doesn't work with llama3.1
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Have you searched existing issues? 🔎
- [X] I have searched and found no existing issues
### Desribe the bug
I am using beropic with llama3.1 for topic modelling. My text is long, so I use doc_length in TextGeneration().
**Error:**
File "/home/bert/lib/python3.11/site-packages/bertopic/representation/_utils.py", line 57, in truncate_document
return truncated_document
^^^^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'truncated_document' where it is not associated with a value
### Reproduction
```python
from bertopic import BERTopic
**here is my code:**
llama3 = TextGeneration(generator, prompt=prompt, nr_docs=4,doc_length=3000)
representation_model = {
"Llama3": llama3
}
topic_model = BERTopic(
embedding_model=embedding_model,
representation_model=representation_model,
umap_model=umap_model,
hdbscan_model=hdbscan_model,
nr_topics = nr_topics,
min_topic_size = 10,
verbose=True,
)
```
### BERTopic Version
0.16.4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in bertopic/representation/_utils.py at truncate_document, then reproduce the failure with TextGeneration(..., doc_length=3000) and the provided BERTopic setup. Done means long documents no longer raise UnboundLocalError and doc_length is applied as intended for llama3.1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100