What explains that similarities are always so high?

I've observed that embeddings generated using SPECTER2 tend to be quite similar, even for conceptually distinct works. While the relative differences between embeddings seem reasonable, I was surprised by how close even unrelated works appear in embedding space.

I noticed this when comparing results obtained via the [Semantic Scholar API](https://github.com/allenai/paper-embedding-public-apis), which seems to yield a broader distribution of distances. I'd appreciate any insights into why this happens and whether it's an expected behavior of SPECTER2.

```python
from transformers import AutoTokenizer
from adapters import AutoAdapterModel
import torch
import sklearn.metrics.pairwise
import numpy as np

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained('allenai/specter2_aug2023refresh_base')

# Load base model
model = AutoAdapterModel.from_pretrained('allenai/specter2_aug2023refresh_base')

# Load the adapter(s) as per the required task, provide an identifier for the adapter in load_as argument and activate it
model.load_adapter("allenai/specter2_aug2023refresh", source="hf", load_as="proximity", set_active=True)

# Create some papers
papers = [
{'title': 'Bioarchaeological and palaeogenomic portrait of two Pompeians', 'abstract': 'This study provides a bioarchaeological and genetic analysis of two individuals who died during the eruption of Mount Vesuvius in 79 AD, offering unprecedented insights into the ancient Roman population.'},
{'title': 'Life tables of annual life expectancy and mortality for companion dogs in the United Kingdom', 'abstract': 'A comprehensive analysis of life expectancy and mortality patterns in different dog breeds, providing valuable data for veterinary medicine and animal welfare.'},
{'title': 'The impact of digital media on children\'s intelligence while controlling for genetic differences in cognition and socioeconomic background', 'abstract': 'This research examines how digital media usage affects cognitive development in children, accounting for genetic and environmental factors.'},
{'title': 'Birdsongs alleviate anxiety and paranoia in healthy participants', 'abstract': 'This study demonstrates the therapeutic potential of natural sounds, specifically birdsong, in reducing anxiety and paranoid thoughts in human subjects.'},
{'title': 'First direct evidence of adult European eels migrating to their breeding place in the Sargasso Sea', 'abstract': 'Groundbreaking research providing the first conclusive evidence of the long-hypothesized migration patterns of European eels to their breeding grounds.'},
{'title': 'The microstructure and the origin of the Venus from Willendorf', 'abstract': 'An analysis of the famous paleolithic Venus figurine, revealing new insights about its material composition and cultural significance.'},
{'title': 'Cooking methods are associated with inflammatory factors, renal function, and other hormones and nutritional biomarkers in older adults', 'abstract': 'This study examines how different cooking methods impact various health markers in elderly populations.'},
{'title': 'Classification of pig calls produced from birth to slaughter according to their emotional valence and context of production', 'abstract': 'A comprehensive analysis of pig vocalizations revealing their emotional and contextual significance throughout their lifecycle.'},
{'title': 'Evidence of an oceanic impact and megatsunami sedimentation in Chryse Planitia, Mars', 'abstract': 'Geological research presenting evidence of an ancient impact event on Mars that may have caused a massive tsunami.'},
{'title': 'Water activated disposable paper battery', 'abstract': 'Development of an innovative, environmentally friendly battery technology activated by water contact.'},
{'title': 'BERT', 'abstract': 'We introduce a new language representation model called BERT'},
{'title': 'Attention is all you need', 'abstract': ' The dominant sequence transduction models are based on complex recurrent or convolutional neural networks'}
]

# Concatenate title and abstract
text_batch = [d['title'] + tokenizer.sep_token + (d.get('abstract') or '') for d in papers]

# Preprocess the input
inputs = tokenizer(text_batch, padding=True, truncation=True, return_tensors="pt", return_token_type_ids=False, max_length=512)

# Forward pass through the model
with torch.no_grad(): # Ensure no gradients are computed
output = model(**inputs)

# Take the first token in the batch as the embedding
embeddings = output.last_hidden_state[:, 0, :]
embeddings = embeddings.detach().cpu().numpy()

# Calculate similarities
cosine_similarities = sklearn.metrics.pairwise.cosine_similarity(embeddings)
np.set_printoptions(precision=4, suppress=True, linewidth=100)
print(np.matrix(cosine_similarities))

```
Output:
```
[[1. 0.8541 0.8171 0.8119 0.8311 0.8796 0.8412 0.8381 0.8454 0.8014 0.8055 0.8139]
[0.8541 1. 0.8 0.8128 0.849 0.8229 0.8578 0.857 0.7919 0.7927 0.8262 0.825 ]
[0.8171 0.8 1. 0.8139 0.7723 0.7729 0.8377 0.8185 0.7687 0.7888 0.8286 0.8251]
[0.8119 0.8128 0.8139 1. 0.803 0.8075 0.8364 0.8736 0.8004 0.7794 0.8331 0.8653]
[0.8311 0.849 0.7723 0.803 1. 0.8088 0.7922 0.8513 0.8237 0.7808 0.7827 0.7846]
[0.8796 0.8229 0.7729 0.8075 0.8088 1. 0.8227 0.8071 0.8423 0.791 0.7935 0.8083]
[0.8412 0.8578 0.8377 0.8364 0.7922 0.8227 1. 0.8305 0.7953 0.8511 0.8028 0.8164]
[0.8381 0.857 0.8185 0.8736 0.8513 0.8071 0.8305 1. 0.7923 0.7837 0.834 0.8524]
[0.8454 0.7919 0.7687 0.8004 0.8237 0.8423 0.7953 0.7923 1. 0.8054 0.7748 0.7949]
[0.8014 0.7927 0.7888 0.7794 0.7808 0.791 0.8511 0.7837 0.8054 1. 0.7821 0.7935]
[0.8055 0.8262 0.8286 0.8331 0.7827 0.7935 0.8028 0.834 0.7748 0.7821 1. 0.898 ]
[0.8139 0.825 0.8251 0.8653 0.7846 0.8083 0.8164 0.8524 0.7949 0.7935 0.898 1. ]]
```