FP16 inaccuracy with Llama-2

FP16 version of llama has strange output.

I could not managed to pinpoint the exac cause yet, but here are my findings:

The following logs contains 5 steps of generation:

MIGRAPHX_TRACE_EVAL=2 python3 txtgen.py -p "Where is Szeged?" -l -s 256 with and without --fp16.
And the timesteps were limited to 5.

llama2-fp16-256-gpu-t5.log
llama2-fp32-256-gpu-t5.log

Note: it can be reproduced with this

The results with FP32:

1) token=13     with prob=18.465343475341797
2) token=13     with prob=12.884403228759766
3) token=29903  with prob=20.014484405517578
4) token=28530  with prob=22.900659561157227
5) token=287    with prob=26.00668716430664

The results with FP16:

1) token=13     with prob=9.03125
2) token=0      with prob=0.0
3) token=18627  with prob=6.38671875
4) token=29901  with prob=9.8359375
5) token=910    with prob=9.671875

Note: 13 is a linebreak.

The problem is at the second step, where the full output for that timestep is 0. After that, the model starts to generate nonsense.