LLaMA/Vicuna 7B fp16 model precision issue on CPU
- Dominant language
- C++
- Stars
- 3.9k
- Forks
- 1k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 47
Description
### What happened?
LLaMA/Vicuna 7B fp16 model is producing all zero results on CPU for both local-sync and local-task.
### Steps to reproduce your issue
1. Download the .mlir from https://storage.googleapis.com/shark-public/vivian/vicuna_fp16.mlir
2. Compile the file with ```iree-compile --iree-input-type=none --iree-hal-target-backends=llvm-cpu --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 --iree-llvmcpu-target-cpu-features=host --iree-llvmcpu-target-triple=x86_64-linux-gnu vicuna_fp16.mlir -o vicuna.vmfb``` Or directly download the compiled vmfb from https://storage.googleapis.com/shark_tank/dan/out16.vmfb
3. Download the inputs: inp1: https://storage.googleapis.com/shark-public/prashant/traced_vicuna/inp1.npy and inp2: https://storage.googleapis.com/shark-public/prashant/traced_vicuna/inp2.npy
4. Run via iree-module: ```iree-run-module --device=local-task --function=forward --input=@inp1.npy --input=@inp2.npy --module=vicuna.vmfb```
Note: it may take a long time to get the results for the first time running the module.
I also tried to use `--iree-flow-break-dispatch=@forward:` on different dispatches, but all generated zero results.
### What component(s) does this issue relate to?
_No response_
### Version information
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.