Best way to take output of transformer model like BERT and run UMAP on it while preserving inverse transformation capability.

Nobody has claimed this yet.

I had this great idea to use transformer sequence to sequence models as an autoencoder.

Unfortunately, I'm getting stuck with a really annoying problem. My encoder outputs vectors of shape (1, *size of sentence*, *number of dimensions in embedding*).
Normally, in a situation like this, I'd be using mean pooling, but that **is not an option for me as I need inverse transformations to faithfully preserve information**.

I know that with BERT, you could use a special token for some tasks, but I want to use the Longformer models and I have not appended any special tokens to my dataset.

Am I going to be stuck using zero vector padding and padding each of my vectors to all be the size of the maximum length sentence? That would create really huge (fixed length) vectors and I can't imagine that would play nicely with UMAP - but at least it's reversible.

All I am trying to do is use transformers to replicate your inverse transform example of hallucinating images, except with hallucinating documents. Why is this so bloody hard? What are my options for this? Has anyone done this before (even with something more simple than a transformer based autoencoder)

P.S. inverse transform not working with sparse outputs is also frustrating for this same reason, as I was really excited about using UMAP on tfidf vectors (and hallucinating key-words rather than whole documents) but this ended up not working for me either on UMAP but did work with LSA/NMF.

No file, test, or entry point is named. Start by reviewing UMAP's inverse-transform behavior for dense and sparse outputs, then compare it with the variable-length transformer representation described here. Done would be a documented reversible approach or a confirmed limitation with clear alternatives.

Written by the indexing model from the issue text.