Distance Matrix: allow basic statistics on distances
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 1.1k
- Avg merge
- 12d 2h
- Merged PRs (30d)
- 1
Description
**What's your use case?**
I performed clustering on a corpus of documents based on t-SNE coordinates. For further analysis, I would like to extract, for each cluster, which other cluster is furthest away, i.e., the most dissimilar. To that end, I computed the average t-SNE x and y coordinates for each cluster using Group By, and then computed Distances based on the coordinates. Based on this, I can create a Distance Matrix like this:

What I would like to extract in an automated way is, for each column, the row ID with the greatest distance and the value of that distance. For other purposes, it may also be useful to get the row ID with the smallest distance with its value, the average distance in each column, etc.
In 'normal' use of the distance matrix, where each row/column represents a data point, it could also be useful to automatically extract for each data point, which other data point is furthest away, how far away it is, etc.
**What's your proposed solution?**
Several options, from most useful to least useful:
1. Create an output with distance statistics (as described below the above image) that can be viewed in Data Table, saved as a file, etc.
2. Create an output with the distances as a 'regular' data table, with labels as row IDs and the same labels as column headers. This would allow some further processing, but I don't see how this gives me the desired results directly. Nevertheless, I can at least consecutively sort descending for each column and get out the max. distances manually.
3. Make the Distance Matrix manipulable the same way as Data Table: allow sorting by clicking on column headers. This would make it easier to manually get the max. distance with associated cluster.
**Are there any alternative solutions?**
- Use Save Distance Matrix, and analyze further using spreadsheet software. Disadvantages: labels have to be added manually, and the top-right half of the matrix is missing, so it has to be created quasi-manually with formulas in the spreadsheet. After that, the max. distances can again be obtained by repeated sorting.
- Use Distance Map, and use a color gradient that makes it relatively easy to pick out the max. values per column (or row). Again, a largely manual approach that isn't without risk of errors.
Contributor guide
Research direction
Start at the Distance Matrix widget and compare its existing behavior with Data Table, Save Distance Matrix, and Distance Map. Trace how matrix labels and distances are exposed, then determine how statistics or a regular table output should represent row IDs and values. Done means the requested per-row or per-column statistics can be inspected or saved and are covered by relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, data-visualization
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100