Context embedding shows anomaly, independent of sentence and token

## Problem
While doing some analysis on the pre-trained SciBert transformer networks, we found that there is an anomaly in the context embedding on index 422. Doing some more tests we found that this anomaly was there, independent of the context or the specific token and position. We think that the distance metrics in the contextualized embedding-space, such as cosine similarity, are heavily dominated by this exploding component.

### scibert-scivocab-uncased

To see the actual contextual representation of words, we infered a sentence using the pretrained model. We performed this using both a TF and PyTorch version of the model to see if the results were similar. Below we can see a crude representation of the embedding space (word + positional embedding) simply as a linear plot of the 768 components of the embedding for the word `linear` in the context of a regular sentence. The values seem nicely distributed around 0 and peaking just a few times just above 2 or -2, which is just as someone would expect the embedding space to look like.

However, if we look at the contextualized embedding, after the last layer of the SciBert model, we find the embedding has turned into this representation below. The representation "explodes" at a particular index (422) very consistently across tokens and sentences.

the token is `linear` is used in the context of the sentence` How are linear regression and gradient descent related, is gradient descent a type of linear regression, and is it similar to ordinary least squares (OLS) and generalized least squares (GLS)?`

### scibert-scivocab-cased

We now performed the same analysis using the cased version of the model, which gives the following results. Here we have the token: `difference` for the sentence:
`Computer Vision: What is the difference between HOG and SIFT feature descriptor?` The exploding index in this embedding is 421, meaning that the anomaly occurs at a different position.

To compared with the uncased model we did the same for the token `linear` for the sentence ` How are linear regression and gradient descent related, is gradient descent a type of linear regression, and is it similar to ordinary least squares (OLS) and generalized least squares (GLS)?` Here we see that the anomaly again is consistent, this also holds for other examples.

From this we see that a similar problem occurs both in the cased and the uncased model, yet the issue is found on a different location from which we figure that something odd is happening in the training process. Again, we have tested this is a behavior in many other tokens and sentences and it's consistent across all circumstances.

Could you explain if this behavior is expected by you and why? And otherwise could you explain what causes this behavior and how it can potentially be overcome?

Thanks in advance!d