iree-org / iree-org/iree

LLaMA/Vicuna 7B fp16 model precision issue on CPU

Open
#14,014 1 comment 0 reactions 1 assignee Claimed by @MaheshRavishankar View on GitHub
bug 🐞
Dominant language
C++
Stars
3.9k
Forks
1k
Avg merge
4d 16h
Merged PRs (30d)
47

Description

### What happened?

LLaMA/Vicuna 7B fp16 model is producing all zero results on CPU for both local-sync and local-task.

### Steps to reproduce your issue

1. Download the .mlir from https://storage.googleapis.com/shark-public/vivian/vicuna_fp16.mlir
2. Compile the file with ```iree-compile --iree-input-type=none --iree-hal-target-backends=llvm-cpu --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 --iree-llvmcpu-target-cpu-features=host --iree-llvmcpu-target-triple=x86_64-linux-gnu vicuna_fp16.mlir -o vicuna.vmfb``` Or directly download the compiled vmfb from https://storage.googleapis.com/shark_tank/dan/out16.vmfb
3. Download the inputs: inp1: https://storage.googleapis.com/shark-public/prashant/traced_vicuna/inp1.npy and inp2: https://storage.googleapis.com/shark-public/prashant/traced_vicuna/inp2.npy
4. Run via iree-module: ```iree-run-module --device=local-task --function=forward --input=@inp1.npy --input=@inp2.npy --module=vicuna.vmfb```

Note: it may take a long time to get the results for the first time running the module.
I also tried to use `--iree-flow-break-dispatch=@forward:` on different dispatches, but all generated zero results.

### What component(s) does this issue relate to?

_No response_

### Version information

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.