Embedding interpretability using SHAP

Nobody has claimed this yet.

Hi,

I've been using umap-learn for a couple of years now, so I just wanted to thank you all for the hard work you put in the package. We find some pretty intriguing insights using it in the chemometrics field.

I'm aware that there are model-agnostic explainers in the field of Machine Learning, such as Shapley Values and LIME, that can help explain how much features contribute to the model's prediction. In the case of Shapley values, these tell you how much each feature contribution deviates from the 'average prediction' (taking the average of all features and predict on that). Here is an ([article](https://towardsdatascience.com/making-sense-of-shapley-values-dc67a8e4c5e8)) explaining how Shapley Values can be applied in the Data Science field despite its origin coming from Game Theory. The beauty of Shapley values is that it is model agnostic, it only asks for an objective function that can project all the features into an output. The objective function being **model.predict()**/**model.transform()** for any model we have built. Though computationally expensive, it can provide insight as to how, say, a Neural Network or SVM dubs which features to be the most influential in explaining the predictions.

I wanted to ask the domain experts if this can be applied to UMAP and if it violates certain assumptions/theory that pertains to the calculation of UMAP embeddings. I've done some testing calculating Shapley Values using the SHAP package on a UMAP model I built. The data is 10 different types of metal concentrations, the class represents the quarry to which the sample came from. I can use Unsupervised UMAP with 1 component and can see class separation on samples from known quarries:

And here is applying the UMAP model to my unknown samples (Class 0):

If I create Shapley values for the model to explain my testing data (samples from unknown quarries), this can tell me notably which metal concentrations contributed the most in determining where the test data resided in the UMAP space. For instance, you'll note Sample 7 is in-between the K and SH group along Embedding 1:

If I look at the Shapley values for Sample 7, I get this result:

where the numbers in the plot pertain to the same units in the Embedding space. So what this plot tells me is that features Ti and K contributed the most towards Sample 7 deviating away from the average location in the 1 dimensional UMAP space. Is this a fair conclusion to make? If not, why? I appreciate you taking the time to look this through.

Best,
Sean Roginski
Eigenvector Research INC.

No repository file, test, or entry point is named. Start by reviewing how UMAP's model.predict() and model.transform() produce embeddings, then compare those assumptions with SHAP's model-agnostic explanations. Done means establishing whether the reported feature contributions are valid and documenting the reasoning or limitations.

Written by the indexing model from the issue text.