Model converts fine to neuralnetwork but produces all nan values for mlprogram

## 🐞Describing the bug
The output of converted Coreml model and original Pytorch model is different. Obvious mismatch is observed. I also notice that there are some similar issues that have been proposed in the past.
[Prediction Mismatch between Pytorch and CoreML #1428](https://github.com/apple/coremltools/issues/1428)
[Wrong output from PyTorch converted model #1486](https://github.com/apple/coremltools/issues/1486)

## To Reproduce
```python
import numpy as np
import torch
from transformers import AutoTokenizer, AutoModel
import coremltools as ct

sentences = ["This is a test."]
tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
model = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2', torchscript=True).eval()
encoded_input = tokenizer(sentences, return_tensors='pt')

traced_model = torch.jit.trace(model, tuple(encoded_input.values()))
scripted_model = torch.jit.script(traced_model)

model = ct.convert(scripted_model, source="pytorch",
inputs=[ct.TensorType(name="input_ids", shape=(ct.RangeDim(), ct.RangeDim()), dtype=np.int32),
ct.TensorType(name="token_type_ids", shape=(ct.RangeDim(), ct.RangeDim()), dtype=np.int32),
ct.TensorType(name="attention_mask", shape=(ct.RangeDim(), ct.RangeDim()), dtype=np.int32)],
convert_to="mlprogram", compute_units=ct.ComputeUnit.CPU_ONLY)
with torch.no_grad():
pt_out = scripted_model(**encoded_input)
cml_inputs = {k: v.to(torch.int32).numpy() for k, v in encoded_input.items()}
pred_coreml = model.predict(cml_inputs)
np.testing.assert_allclose(pt_out[0].detach().numpy(), pred_coreml["hidden_states"], atol=1e-5, rtol=1e-4)
```

## System environment (please complete the following information):
- coremltools version: 6.2
- OS: MacOS 13.2
- torch: 1.12.1
- numpy: 1.24.1
- transformer: 4.25.1