iree-org / iree-org/iree

[GPU][Codegen] SDXL value mismatch without spec file on MI300

Open
#19,984 1 comment 0 reactions 0 assignees View on GitHub
bug 🐞
Dominant language
C++
Stars
3.9k
Forks
1k
Avg merge
4d 16h
Merged PRs (30d)
47

Description

### What happened?

`test_unet.py` will run into a regression failure when exercising it locally on int8_fp16 mixed precision tests if not careful about it. The problem here is two-fold:
- compiler team needs to figure out and fix the regression without spec file
- usability of the script: assume a user of the script just want to reproduce the github action result, make it take the default spec location instead of relying on an environment variable. I can assign you to this subtask. CC @geomin12

### Steps to reproduce your issue

### To reproduce the failure

1. Use a MI300 machine, build iree and make sure iree-compile and iree-run-module are accessible from PATH
2. Run standalone pytest without fully mimicking the CI steps, basically:
```bash
cd iree
pytest ./experimental/regression_suite/shark-test-suite-models/sdxl/test_unet.py -k int8_fp16_rocm -rpfE --log-cli-level=info --timeout=600 --durations=0
```
3. Observe the failure and corresponding log
> input = None, capture_output = True, timeout = None, check = True
popenargs = (['iree-run-module', '--device=hip', '--module=/root/iree/sdxl_punet_int8_fp16_vmfbs/punet_fp16.rocm_gfx942.vmfb', '--...fp16/punet_weights.irpa', '--input=1x4x128x128xf16=@/root/iree/artifacts/sdxl_punet_int8/inference_input.0.bin', ...],) kwargs = {'cwd': PosixPath('/root/iree/sdxl_punet_int8_fp16_vmfbs'), 'stderr': -1, 'stdout': -1}, process =
stdout = b'EXEC @main\n[FAILED] result[0]: element at index 0 (0.0032959) does not match the expected (0.0914307); expected tha......][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...]]]\n' stderr = b'', retcode = 1

### To fix it

Comment out below lines

https://github.com/iree-org/iree/blob/8fab35ce603ecd6cef5c90cb62c0115b432aa66d/experimental/regression_suite/shark-test-suite-models/sdxl/test_unet.py#L294-L304

Replace it with the right spec location, i.e, in my docker container, it is:

```python
INT8_PUNET_FLAGS.append(f"--iree-codegen-transform-dialect-library=/root/iree/build_tools/pkgci/external_test_suite/attention_and_matmul_spec_punet_mi300.mlir")
```

Then run pytest again, observe the unit tests pass with matching values.

### What component(s) does this issue relate to?

_No response_

### Version information

latest: 5767be368fc8a8a8a7f1fcfb2da27b40566bb7ed

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with experimental/regression_suite/shark-test-suite-models/sdxl/test_unet.py, especially lines 294-304 and INT8_PUNET_FLAGS, and inspect how the script obtains its spec path. Run the provided pytest command on an MI300 after building IREE and making the tools available on PATH; done means standalone int8_fp16_rocm tests use the default spec location and pass with matching values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.