[GPU][Codegen] SDXL value mismatch without spec file on MI300
- Dominant language
- C++
- Stars
- 3.9k
- Forks
- 1k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 47
Description
### What happened?
`test_unet.py` will run into a regression failure when exercising it locally on int8_fp16 mixed precision tests if not careful about it. The problem here is two-fold:
- compiler team needs to figure out and fix the regression without spec file
- usability of the script: assume a user of the script just want to reproduce the github action result, make it take the default spec location instead of relying on an environment variable. I can assign you to this subtask. CC @geomin12
### Steps to reproduce your issue
### To reproduce the failure
1. Use a MI300 machine, build iree and make sure iree-compile and iree-run-module are accessible from PATH
2. Run standalone pytest without fully mimicking the CI steps, basically:
```bash
cd iree
pytest ./experimental/regression_suite/shark-test-suite-models/sdxl/test_unet.py -k int8_fp16_rocm -rpfE --log-cli-level=info --timeout=600 --durations=0
```
3. Observe the failure and corresponding log
> input = None, capture_output = True, timeout = None, check = True
popenargs = (['iree-run-module', '--device=hip', '--module=/root/iree/sdxl_punet_int8_fp16_vmfbs/punet_fp16.rocm_gfx942.vmfb', '--...fp16/punet_weights.irpa', '--input=1x4x128x128xf16=@/root/iree/artifacts/sdxl_punet_int8/inference_input.0.bin', ...],) kwargs = {'cwd': PosixPath('/root/iree/sdxl_punet_int8_fp16_vmfbs'), 'stderr': -1, 'stdout': -1}, process =
stdout = b'EXEC @main\n[FAILED] result[0]: element at index 0 (0.0032959) does not match the expected (0.0914307); expected tha......][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...]]]\n' stderr = b'', retcode = 1
### To fix it
Comment out below lines
https://github.com/iree-org/iree/blob/8fab35ce603ecd6cef5c90cb62c0115b432aa66d/experimental/regression_suite/shark-test-suite-models/sdxl/test_unet.py#L294-L304
Replace it with the right spec location, i.e, in my docker container, it is:
```python
INT8_PUNET_FLAGS.append(f"--iree-codegen-transform-dialect-library=/root/iree/build_tools/pkgci/external_test_suite/attention_and_matmul_spec_punet_mi300.mlir")
```
Then run pytest again, observe the unit tests pass with matching values.
### What component(s) does this issue relate to?
_No response_
### Version information
latest: 5767be368fc8a8a8a7f1fcfb2da27b40566bb7ed
### Additional context
_No response_
Contributor guide
Research direction
Start with experimental/regression_suite/shark-test-suite-models/sdxl/test_unet.py, especially lines 294-304 and INT8_PUNET_FLAGS, and inspect how the script obtains its spec path. Run the provided pytest command on an MI300 after building IREE and making the tools available on PATH; done means standalone int8_fp16_rocm tests use the default spec location and pass with matching values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- compilers, machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100