intel / intel/xFasterTransformer
[output issue] found mistakes in llama-3-70b output by bf16_int4 during benchmark
- Dominant language
- C++
- Stars
- 435
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
**weights**: Meta-Llama-3-70B-Instruct
**precision**: bf16_int4 (v.s. bf16)
**version**: 1.6.0
**hardware**: 2S-SPR9468 (Quadrant/Flat)
**system**: Ubuntu22.04LTS container (latest XFT image)
**kernel**: 5.17.3
**command**:
> bf16 precision:
```bash
bash run_benchmark.sh -m llama-3-70b -d bf16 -s 2 -bs 1 -in 1024 -out 128 -i 1
```
>bf16_int4:
```bash
bash run_benchmark.sh -m llama-3-70b -d bf16_int4 -s 2 -bs 1 -in 1024 -out 128 -i 1
```
**issue**:
> on bf16 precision, output is valid:
>on bf16_int4 precision, output is invalid:
Contributor guide
Assessment
This issue has not been assessed yet.