Duplicate Document Entries in Documents from get_document_info(corpus)

Nobody has claimed this yet.

Hello!
I am working on analyzing research funding trends using data collected from NIH ExPORTER.

My team and I have run into a problem where there are no duplicate documents in our raw data, as in there are not repeated ABSTRACT, APPLICATION_ID, or PI_names, but documents extracted from the BERTopic results using get_document_info(corpus) has repeated documents with the same exact ABSTRACT, APPLICATION_ID, and PI_names. Even the dates of budget start and end are the same.

I don't think the problem lies in preprocessing, as our corpus creation is rather simple:
```
def clean_text(text):
text = text.lower().strip() # Lowercase
text = re.sub(r'\s+', ' ', text) # Remove extra spaces
text = re.sub(r'[^\w\s]', '', text) # Remove punctuation
text = re.sub(r'\b\d+\b', '', text) # Remove standalone numbers
return text

data['CLEANED_ABSTRACT'] = data['ABSTRACT_TEXT'].astype(str).apply(clean_text)
corpus = data['CLEANED_ABSTRACT'].astype(str).tolist()
```

There shouldn't be repeated documents, since there are no resubmissions or renewals(which would mean a different start date on the budget, but the same abstract and PI names, but there aren't any repeats in our raw data file).

The repeated documents only appear in doc_info that was retrieved via get_document_info(corpus). We're thinking that duplicates maybe appearing because some documents are assigned 2 or more topics at once, and replacing some of the other documents assigned to those same topics with lower probabilities. However, we don't see these repeat documents in different topics, only in the same ones. And in our manual validation where we picked 15 documents for 15 topics, we will get anywhere from 2~8 repeated documents per topic.

Any idea is appreciated. Please let me know if more information is needed to answer this question. Thank you.

Start by reproducing the reported get_document_info(corpus) call with the NIH ExPORTER corpus and compare the returned rows with the original corpus entries, including their topics and document indices. Determine whether the repeated rows are introduced by get_document_info or are already present in the input, then document the reproduction and expected one-row-per-document behavior.

Written by the indexing model from the issue text.