BERT models not working as expected for cross-lingual data

We are testing BERT on a cross lingual dataset with different permutations. Either both the sentences are in English or both of them in Hindi or one of them is English and another one is Hindi explained in the attached file.

Our observation is as follows -
1. Bert-multi-base-cased without any fine tuning on "Hindi" gives every sentence a high score and we think that is randomly generated.
2. When we fine tuned " Bert-multi-base-cased" using XNLI "Hindi" corpus then we start getting good results. If both the sentences are in the same language, the results are as expected and correct. Which also means that the model has individually learnt both the languages. But the moment we evaluate two different languages together (in our case Hindi & English) it starts giving low score for everything.
One of the explanations could be that since we only fine tuned on XNLI Hindi the vector space of English and Hindi became different and hence the scores are incorrect.
So, to avoid that we tried the next approach---
3. We fine tuned "Bert-multi-base-cased" on XNLI Hindi and English together by appending both the datasets. But we didn't see any difference. The result was similar to the previous one.

We are not quite sure if our approach is correct. Could you please share more information on this ?
Thanks.
[bert_models_benchmarking.xlsx](https://github.com/UKPLab/sentence-transformers/files/4138142/bert_models_benchmarking.xlsx)

Start by reviewing the attached bert_models_benchmarking.xlsx and reconstructing the English, Hindi, and mixed-language evaluation permutations described in the issue. Compare the untuned model with the Hindi-only and combined XNLI fine-tuning setups; done would require a reproducible diagnosis of the reported cross-lingual scores and a clearly scoped follow-up change.

Written by the indexing model from the issue text.