Has someone tried adding an Auto Encoder on top of S-BERT?

Hi all,
First of all, to the authors great work on the paper and repo :)

This repo might or might not be the best place to post it, but was wondering if someone had tried adding an Auto Encoder on top of the output of S-BERT?

Something like a VAE?

The intuition is that forcing an AE on top of S-BERT forces it to reduce dimension. With an MSE loss, hopefully the vector in reduced dimension captures all information.

The benefits to name a few are :
1. Reduced dimension, means that vector search would be faster, all other downstream compute would be faster.
2. Something like VAE can even also open up new areas of research where we can see that some latent variables contribute to **funny**, whereas other contributes to **political**..

And hopefully a VAE can generalise to unseen data.

I tried a PCA on some inhouse dataset, and saw
- 256 vectors explain 99.02% variance
- 512 vectors explain 99.97% variance

Saw an SSE of 525, 100 respectively on Test Set of 100 examples, which might not be that bad given that each of the 100 examples has 768 dimensions.

The results are somewhat motivating, and I was wondering if this topic in particular interests someone. Or if someone knows of active/past research in this field.

Would be happy to contribute to it in whichever way possible.

No files, tests, or entry points are named. Start by locating the S-BERT output and training workflow, then evaluate an autoencoder or VAE against the PCA results described in the issue; done should include reduced-dimensional embeddings and evidence of reconstruction and downstream vector-search performance.

Written by the indexing model from the issue text.