Identifying more than one topic in a large document.

Nobody has claimed this yet.

Hi there,

I have built a topic model that identifies topics from political speech data over a longterm period. Most of the 'documents' are a few sentences long, with a few outliers that are longer documents that presumably contain more than one topic. However, I found the default bertopic pipeline produced good topics without having to break these minority instances of long documents down into shorter documents.

That being said, at this point, I have a specific long document that I believe has mixed membership of topics, but of course has only been labelled as having one topic by the model. Does it make sense to break this mixed-membership document down into smaller chunks (let's say paragraphs) and then predict the topic for each paragraph?

Also, I am looking to compare to sets of documents within the corpus at a high level. I.e. measuring similarity between topics occurring in the first set to the second set. I was wondering what the best way to do this might be?

Thank you.

Review the default BERTopic pipeline and its prediction step for long documents. Determine whether paragraph-level topic assignment and comparison of topic sets are supported, then document the recommended approach and its limitations.

Written by the indexing model from the issue text.