biolab / biolab/orange3

Distance Matrix: allow basic statistics on distances

Open
#6,556 2 comments 0 reactions 0 assignees View on GitHub
meal wish
Dominant language
Python
Stars
5.7k
Forks
1.1k
Avg merge
12d 2h
Merged PRs (30d)
1

Description

**What's your use case?**
I performed clustering on a corpus of documents based on t-SNE coordinates. For further analysis, I would like to extract, for each cluster, which other cluster is furthest away, i.e., the most dissimilar. To that end, I computed the average t-SNE x and y coordinates for each cluster using Group By, and then computed Distances based on the coordinates. Based on this, I can create a Distance Matrix like this:
![image](https://github.com/biolab/orange3/assets/55989717/5a94abb0-c7e1-403a-a6e2-9693f7016bcf)
What I would like to extract in an automated way is, for each column, the row ID with the greatest distance and the value of that distance. For other purposes, it may also be useful to get the row ID with the smallest distance with its value, the average distance in each column, etc.
In 'normal' use of the distance matrix, where each row/column represents a data point, it could also be useful to automatically extract for each data point, which other data point is furthest away, how far away it is, etc.

**What's your proposed solution?**
Several options, from most useful to least useful:

1. Create an output with distance statistics (as described below the above image) that can be viewed in Data Table, saved as a file, etc.
2. Create an output with the distances as a 'regular' data table, with labels as row IDs and the same labels as column headers. This would allow some further processing, but I don't see how this gives me the desired results directly. Nevertheless, I can at least consecutively sort descending for each column and get out the max. distances manually.
3. Make the Distance Matrix manipulable the same way as Data Table: allow sorting by clicking on column headers. This would make it easier to manually get the max. distance with associated cluster.

**Are there any alternative solutions?**

- Use Save Distance Matrix, and analyze further using spreadsheet software. Disadvantages: labels have to be added manually, and the top-right half of the matrix is missing, so it has to be created quasi-manually with formulas in the spreadsheet. After that, the max. distances can again be obtained by repeated sorting.
- Use Distance Map, and use a color gradient that makes it relatively easy to pick out the max. values per column (or row). Again, a largely manual approach that isn't without risk of errors.

Contributor guide

Open the contributing guide

Research direction

Start at the Distance Matrix widget and compare its existing behavior with Data Table, Save Distance Matrix, and Distance Map. Trace how matrix labels and distances are exposed, then determine how statistics or a regular table output should represent row IDs and values. Done means the requested per-row or per-column statistics can be inspected or saved and are covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, data-visualization
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.