Setting up an end-to-end system with BERTopic efficiently.

Nobody has claimed this yet.

Hi Maarten,

Thanks for developing such a modular topic modeling setup and the awesome library!
I have been trying to setup a system with BERTopic which performs the following steps:

1. Given a set of starting documents, train a BERTopic model to get relevant topics and create a `{topic_id: topic}` map
2. Given new documents, perform inference on them to assign them to existing _topic_ids_
3. Train with only the new documents to assign them to previous _topic_ids_ and create new _topic_ids_

My questions are as follows:

- I have some documents that are larger than the length of what `sentence transformers` accept and would like to chunk them. I am not sure how to maintain an `id` for the document with BERTopic such that after training/inference I can associate multiple topics and thus _topic_ids_ with the document.
- Do you have recommendations for training the model in step 3 such that we do not need to re-train with documents used in _step_1_?
- Any other suggestions you might have for the steps outlined above!

Start by reviewing the BERTopic workflow described in the issue, including document chunking, inference against existing topic IDs, and incremental training. The issue is complete when the project provides documented guidance for preserving document associations and updating a model without retraining on the original documents.

Written by the indexing model from the issue text.