spcl / spcl/dace

SDFG Invalid / Incorrect if 2GPU is Applied Twice

Open
#2,003 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
593
Forks
163
Avg merge
2d 23h
Merged PRs (30d)
60

Description

If 2GPU is applied twice, the resulting SDFG is often invalid, sometimes passes `.validate()` but crashes after compiling.
This bug happens on `crc16` in NPBench repository.

To reproduce you can run:
```
import dace
import copy

sdfg = dace.SDFG.from_file("crc16_strict.sdfgz")
sdfg2 = copy.deepcopy(sdfg)
sdfg3 = copy.deepcopy(sdfg)
sdfg4 = copy.deepcopy(sdfg)

# OK
sdfg.simplify()
sdfg.apply_gpu_transformations()
sdfg.validate()
sdfg.compile()

# OK
for arr_name, arr in sdfg2.arrays.items():
if not arr.transient:
arr.storage = dace.dtypes.StorageType.GPU_Global
sdfg2.apply_gpu_transformations(validate=False)
sdfg2.save("crc16_repr_gpu.sdfgz", compress=True)
sdfg2.validate()
sdfg2.compile()

# OK
for arr_name, arr in sdfg3.arrays.items():
if not arr.transient:
arr.storage = dace.dtypes.StorageType.GPU_Global
sdfg3.auto_optimize(device=dace.dtypes.DeviceType.GPU, validate=False)
sdfg3.save("crc16_repr_aopt.sdfgz", compress=True)
sdfg3.validate()
sdfg3.compile()

# Not OK
for arr_name, arr in sdfg4.arrays.items():
if not arr.transient:
arr.storage = dace.dtypes.StorageType.GPU_Global
sdfg4.apply_gpu_transformations()
sdfg4.auto_optimize(device=dace.dtypes.DeviceType.GPU, validate=False)
sdfg4.save("crc16_repr_aopt_double.sdfgz", compress=True)
sdfg4.validate()
sdfg4.compile()
```

I attach the SDFG files needed to reproduce with a `.txt` suffix such that github uploads them.

You can check the invalid SDFG from NPBench: `crc16_invalid_from_npbench.sdfgz`
and use `crc16_strict.sdgz` to reproduce.

Error is this: First version moves GPU data to HOST but the second call moves the HOST transient to GPU without checking anything.
```
File "/home/primrose/Work/npbench/repr_crc16.py", line 35, in
if not arr.transient:
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/primrose/Work/dace/dace/sdfg/sdfg.py", line 2564, in auto_optimize
auto_optimize(self, device, validate, validate_all, symbols, use_gpu_storage)
File "/home/primrose/Work/dace/dace/transformation/auto/auto_optimize.py", line 606, in auto_optimize
sdfg.apply_gpu_transformations()
File "/home/primrose/Work/dace/dace/sdfg/sdfg.py", line 2784, in apply_gpu_transformations
self.apply_transformations(GPUTransformSDFG,
File "/home/primrose/Work/dace/dace/sdfg/sdfg.py", line 2653, in apply_transformations
results = pazz.apply_pass(self, {})
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/primrose/Work/dace/dace/transformation/passes/pattern_matching.py", line 129, in apply_pass
result = match.apply(graph, tcfg.sdfg)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/primrose/Work/dace/dace/transformation/interstate/gpu_transform_sdfg.py", line 625, in apply
sdfg.simplify()
File "/home/primrose/Work/dace/dace/sdfg/sdfg.py", line 2532, in simplify
pass_options=options).apply_pass(self, {})
^^^^^^^^^^^^^^^^^^^^
File "/home/primrose/Work/dace/dace/transformation/passes/simplify.py", line 165, in apply_pass
sdfg.validate()
File "/home/primrose/Work/dace/dace/sdfg/sdfg.py", line 2497, in validate
validate_sdfg(self, references, **context)
File "/home/primrose/Work/dace/dace/sdfg/validation.py", line 336, in validate_sdfg
validate_control_flow_region(sdfg, sdfg, initialized_transients, symbols, references, **context)
File "/home/primrose/Work/dace/dace/sdfg/validation.py", line 153, in validate_control_flow_region
validate_control_flow_region(sdfg, edge.dst, initialized_transients, lsyms, references, **context)
File "/home/primrose/Work/dace/dace/sdfg/validation.py", line 120, in validate_control_flow_region
raise InvalidSDFGInterstateEdgeError(
dace.sdfg.validation.InvalidSDFGInterstateEdgeError: Trying to read an inaccessible data container "host_data" (Storage: StorageType.GPU_Global) in host code interstate edge (at edge "crc=__tmp13" (block_4 -> BinOp_27)
Invalid SDFG saved for inspection in /home/primrose/Work/npbench/_dacegraphs/invalid.sdfgz
```

Should we have a flag that marks SDFGs as offloaded and avoid running 2GPU again on the alread-offloaded SDFG? I am asking as to aid in the design of the new 2GPU transformation I will writing off as a thesis.
Can we assume the user is smart enough to not call 2GPU twice? (And fix this in NPBench).

I think this should be fixed in both locations. We should not call 2GPU again, but also have a check not to offload an SDFG twice.

[crc16_invalid_from_npbench.sdfgz.txt](https://github.com/user-attachments/files/20100418/crc16_invalid_from_npbench.sdfgz.txt)

[crc16_strict.sdfgz.txt](https://github.com/user-attachments/files/20100420/crc16_strict.sdfgz.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.