Get index of docs plotted using sample param in visualize_documents and "smart" way to reduce data points?

Nobody has claimed this yet.

Hi,

Is it possible to get the index of documents plotted when using the sample parameter in visualize_documents? As far as I understand it, if, say, sample=0.1, from each topic a random 10% of docs is sampled to visualize in the plot. Would help greatly to see which docs specifically, as we use GPT or other LLM API to summarize each doc and show this as hover in the plot. We only want to summarize the docs actually plotted.

edit: Generally, we have been thinking about how to decrease the number of data points plotted, preferably in a "smart" way, for the same reason of computational and cost constraints when using some LLM API to interpret the topic labels or docs themselves. I've used MMR for topic fine-tuning, so filtering terms with high marginal relevance, it definitely improved the output. Is there something similar for documents themselves? Plotting only the documents that are relevant or contain many highly marginal relevant terms? Anyone experience with this?

Would love to hear about it. Regards and thanks,
Arne

Start at the visualize_documents entry point and trace how the sample parameter selects documents for plotting. Define how plotted document indices would be exposed so external summaries can target them, then evaluate whether the separate request for a smarter reduction strategy is in scope or needs its own design.

Written by the indexing model from the issue text.