SDFG Invalid / Incorrect if 2GPU is Applied Twice
- Dominant language
- Python
- Stars
- 593
- Forks
- 163
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 60
Description
If 2GPU is applied twice, the resulting SDFG is often invalid, sometimes passes `.validate()` but crashes after compiling.
This bug happens on `crc16` in NPBench repository.
To reproduce you can run:
```
import dace
import copy
sdfg = dace.SDFG.from_file("crc16_strict.sdfgz")
sdfg2 = copy.deepcopy(sdfg)
sdfg3 = copy.deepcopy(sdfg)
sdfg4 = copy.deepcopy(sdfg)
# OK
sdfg.simplify()
sdfg.apply_gpu_transformations()
sdfg.validate()
sdfg.compile()
# OK
for arr_name, arr in sdfg2.arrays.items():
if not arr.transient:
arr.storage = dace.dtypes.StorageType.GPU_Global
sdfg2.apply_gpu_transformations(validate=False)
sdfg2.save("crc16_repr_gpu.sdfgz", compress=True)
sdfg2.validate()
sdfg2.compile()
# OK
for arr_name, arr in sdfg3.arrays.items():
if not arr.transient:
arr.storage = dace.dtypes.StorageType.GPU_Global
sdfg3.auto_optimize(device=dace.dtypes.DeviceType.GPU, validate=False)
sdfg3.save("crc16_repr_aopt.sdfgz", compress=True)
sdfg3.validate()
sdfg3.compile()
# Not OK
for arr_name, arr in sdfg4.arrays.items():
if not arr.transient:
arr.storage = dace.dtypes.StorageType.GPU_Global
sdfg4.apply_gpu_transformations()
sdfg4.auto_optimize(device=dace.dtypes.DeviceType.GPU, validate=False)
sdfg4.save("crc16_repr_aopt_double.sdfgz", compress=True)
sdfg4.validate()
sdfg4.compile()
```
I attach the SDFG files needed to reproduce with a `.txt` suffix such that github uploads them.
You can check the invalid SDFG from NPBench: `crc16_invalid_from_npbench.sdfgz`
and use `crc16_strict.sdgz` to reproduce.
Error is this: First version moves GPU data to HOST but the second call moves the HOST transient to GPU without checking anything.
```
File "/home/primrose/Work/npbench/repr_crc16.py", line 35, in
if not arr.transient:
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/primrose/Work/dace/dace/sdfg/sdfg.py", line 2564, in auto_optimize
auto_optimize(self, device, validate, validate_all, symbols, use_gpu_storage)
File "/home/primrose/Work/dace/dace/transformation/auto/auto_optimize.py", line 606, in auto_optimize
sdfg.apply_gpu_transformations()
File "/home/primrose/Work/dace/dace/sdfg/sdfg.py", line 2784, in apply_gpu_transformations
self.apply_transformations(GPUTransformSDFG,
File "/home/primrose/Work/dace/dace/sdfg/sdfg.py", line 2653, in apply_transformations
results = pazz.apply_pass(self, {})
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/primrose/Work/dace/dace/transformation/passes/pattern_matching.py", line 129, in apply_pass
result = match.apply(graph, tcfg.sdfg)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/primrose/Work/dace/dace/transformation/interstate/gpu_transform_sdfg.py", line 625, in apply
sdfg.simplify()
File "/home/primrose/Work/dace/dace/sdfg/sdfg.py", line 2532, in simplify
pass_options=options).apply_pass(self, {})
^^^^^^^^^^^^^^^^^^^^
File "/home/primrose/Work/dace/dace/transformation/passes/simplify.py", line 165, in apply_pass
sdfg.validate()
File "/home/primrose/Work/dace/dace/sdfg/sdfg.py", line 2497, in validate
validate_sdfg(self, references, **context)
File "/home/primrose/Work/dace/dace/sdfg/validation.py", line 336, in validate_sdfg
validate_control_flow_region(sdfg, sdfg, initialized_transients, symbols, references, **context)
File "/home/primrose/Work/dace/dace/sdfg/validation.py", line 153, in validate_control_flow_region
validate_control_flow_region(sdfg, edge.dst, initialized_transients, lsyms, references, **context)
File "/home/primrose/Work/dace/dace/sdfg/validation.py", line 120, in validate_control_flow_region
raise InvalidSDFGInterstateEdgeError(
dace.sdfg.validation.InvalidSDFGInterstateEdgeError: Trying to read an inaccessible data container "host_data" (Storage: StorageType.GPU_Global) in host code interstate edge (at edge "crc=__tmp13" (block_4 -> BinOp_27)
Invalid SDFG saved for inspection in /home/primrose/Work/npbench/_dacegraphs/invalid.sdfgz
```
Should we have a flag that marks SDFGs as offloaded and avoid running 2GPU again on the alread-offloaded SDFG? I am asking as to aid in the design of the new 2GPU transformation I will writing off as a thesis.
Can we assume the user is smart enough to not call 2GPU twice? (And fix this in NPBench).
I think this should be fixed in both locations. We should not call 2GPU again, but also have a check not to offload an SDFG twice.
[crc16_invalid_from_npbench.sdfgz.txt](https://github.com/user-attachments/files/20100418/crc16_invalid_from_npbench.sdfgz.txt)
[crc16_strict.sdfgz.txt](https://github.com/user-attachments/files/20100420/crc16_strict.sdfgz.txt)
Contributor guide
Assessment
This issue has not been assessed yet.